Repository: flairNLP/flair

Issue:
Multilingual Language Models
Hi,

I trained language models for 16 languages on Wikipedia dumps + OPUS, that can be integrated into `flair` :) This is the result of ~ 2 months work.

# Language models

Training data are (a) recent Wikipedia dump and (b) corpora from OPUS. Training was done for one epoch over the full training corpus.

| Language (Code) | Tokens (training) | Forward ppl | Backward ppl
| --------------- | ----------------- | ----------- | ------------
| Arabic (ar)     | 736,512,400       | 3.39        | 3.45
| Bulgarian (bg)  | 111,336,781       | 2.46        | 2.47
| Czech (cs)      | 442,892,103       | 2.89        | 2.90
| Danish (da)     | 325,816,384       | 2.62        | 2.68
| Basque (eu)     | 36,424,055        | 2.64        | 2.31
| Persian (fa)    | 146,619,206       | 3.68        | 3.66
| Finnish (fi)    | 427,194,262       | 2.63        | 2.65
| Hebrew (he)     | 502,949,245       | 3.84        | 3.87
| Hindi (hi)      | 28,936,996        | 2.87        | 2.86
| Croatian (hr)   | 625,084,958       | 3.13        | 3.20
| Indonesian (id) | 174,467,241       | 2.80        | 2.74
| Italian (it) | 1,549,430,560 | 2.62 | 2.63
| Dutch (nl)      | 1,275,949,108     | 2.43        | 2.55
| Norwegian (no)  | 156,076,225       | 3.01        | 3.01
| Polish (pl)     | 1,428,604,528     | 2.95        | 2.84
| Slovenian (sl)  | 419,744,423       | 2.88        | 2.91
| Swedish (sv)    | 671,922,632       | 6.82        | 2.25

Download links:

```bash
wget https://schweter.eu/cloud/flair-lms/lm-ar-opus-large-forward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-ar-opus-large-backward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-bg-opus-large-forward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-bg-opus-large-backward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-cs-opus-large-forward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-cs-opus-large-backward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-da-opus-large-forward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-da-opus-large-backward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-eu-opus-large-forward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-eu-opus-large-backward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-fa-opus-large-forward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-fa-opus-large-backward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-fi-opus-large-forward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-fi-opus-large-backward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-he-opus-large-forward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-he-opus-large-backward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-hi-opus-large-forward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-hi-opus-large-backward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-hr-opus-large-forward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-hr-opus-large-backward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-id-opus-large-forward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-id-opus-large-backward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-it-opus-large-forward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-it-opus-large-backward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-nl-opus-large-forward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-nl-opus-large-backward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-no-opus-large-forward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-no-opus-large-backward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-pl-opus-large-forward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-pl-opus-large-backward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-sl-opus-large-forward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-sl-opus-large-backward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-sv-opus-large-forward-v0.1.pt
wget https://schweter.eu/cloud/flair-lms/lm-sv-opus-large-backward-v0.1.pt
```

Hyperparameters:

| Parameter         | Value
| ----------------- | -----
| `hidden_size`     | 2048
| `nlayers`         | 1
| `sequence_length` | 250
| `mini_batch_size` | 100

Instead of using `common_chars`, all characters (from the training corpus) are used as vocabulary for language model training.

# PoS Tagging on Universal Dependencies (v1.2)

To test the new language models on a downstream task, results for PoS tagging on Universal Dependencies (v1.2) are reported (with comparisons to other papers).

| Language (Code) | [Yu et. al (2017)](https://arxiv.org/abs/1706.01723) | [Plank et. al (2016)](https://arxiv.org/abs/1604.05529) | [Yasunaga et. al (2017)](https://arxiv.org/abs/1711.04903) | Flair     | Δ 
| --------------- | ---------------------------------------------------- | ------------------------------------------------------- | ---------------------------------------------------------- | --------- | -------
| Arabic (ar)     | **99.00**                                            | 98.91                                                   | n.a.                                                       | 98.86     | -0.14
| Bulgarian (bg)  | 98.20                                                | 98.23                                                   | 98.53                                                      | **99.18** | 0.65
| Czech (cs)      | 98.79                                                | 98.24                                                   | 98.81                                                      | **99.14** | 0.33
| Danish (da)     | 95.92                                                | 96.35                                                   | 96.74                                                      | **98.48** | 1.74🔥
| Basque (eu)     | 94.94                                                | 95.51                                                   | 94.71                                                      | **97.30** | 1.79🔥
| Persian (fa)    | 97.12                                                | 97.60                                                   | 97.51                                                      | **98.15** | 0.55
| Finnish (fi)    | 95.31                                                | 95.85                                                   | 95.40                                                      | **98.11** | 2.26🔥
| Hebrew (he)     | 96.04                                                | 96.96                                                   | 97.43                                                      | **97.67** | 0.24
| Hindi (hi)      | 96.96                                                | 97.10                                                   | 97.21                                                      | **97.85** | 0.64
| Croatian (hr)   | 95.05                                                | 96.82                                                   | 96.32                                                      | **97.43** | 0.61
| Indonesian (id) | 93.44                                                | 93.41                                                   | **94.03**                                                  | 93.85     | -0.18
| Dutch (nl)      | 93.11                                                | 93.82                                                   | 93.09                                                      | **94.03** | 0.21
| Norwegian (no)  | 97.65                                                | 98.06                                                   | 98.08                                                      | **98.73** | 0.65
| Polish (pl)     | 96.83                                                | 97.63                                                   | 97.57                                                      | **98.81** | 1.18🔥
| Slovenian (sl)  | 97.16                                                | 96.97                                                   | 98.11                                                      | **99.02** | 0.91
| Swedish (sv)    | 96.28                                                | 96.69                                                   | 96.70                                                      | **98.54** | 1.84🔥

Hyperparameters:

| Parameter              | Value
| ---------------------- | -----
| `hidden_size`          | `512`
| `learning_rate`        | `0.1`
| `mini_batch_size`      | `8`
| `max_epochs`           | `500`

Results on Universal Dependencies show new SOTA, except for Arabic and Indonesian.


Discussion:
Hi @stefan-it this is awesome, looking forward to integrating this!! 

Is there a paper on your results?
@alanakbik I was just thinking about submitting a workshop paper. But I just got a ACL rejection, because we just included the supplementary material inside the paper... this is a really demotivating factor 🤣
Yeah I think we all know the feeling :/ But there's always another conference on the horizon :) Any plans on putting the paper on archiv?  
Thank you very much @stefan-it . Coming here from #179 . If you don't mind can you please answer the following questions regarding Arabic LM:
1. How many epochs did you train for?
2. What was your initial learning rate? Can you provide a rough idea of how your learning rate dropped over the epochs?

The reason I ask is that I am training an LM over a 1.5B word Arabic corpus. And I would like to get some pointers on when to stop training. It's been 2 days on a K80 and after 1 epoch, I am looking at a perplexity of roughly 3.6 and the learning rate has dropped to 5.

Maybe, even @alanakbik could share his experience over training over huge corpora.

Thanks
Zeeshan

Hello @zeeshansayyed we generally train for about 2 weeks. One thing I note is that your learning rate has already annealed to 5 after 2 days which indicates that your patience may be too low. Try doubling the patience so that you learn with a learning rate of 20 for a few more days. 
Yes. On the 4th day, the learning rate fell to 1.25. In the code, the default patience seems to be at 10. But elsewhere on the internet, I have seen it be 3 or 5. Do you think a 'patience' of 20 would do the trick? I will stop and restart training.
Yes, 20 will be better. You could even go higher if you have enough time (the higher the patience the longer it trains).
Hi @zeeshansayyed :)

to answer your questions: I trained all models for one epoch. Initial learning rate was *20*. In my experiments I used a training corpus split size < 20, so that the learning rate never decreased.

Resolving patch:
diff --git a/flair/data.py b/flair/data.py
--- a/flair/data.py
+++ b/flair/data.py
@@ -371,7 +371,7 @@ def __init__(
                     self.add_token(token)
 
         # log a warning if the dataset is empty
-        if not text:
+        if text == "":
             log.warn(
                 "ACHTUNG: An empty Sentence was created! Are there empty strings in your dataset?"
             )
diff --git a/flair/embeddings.py b/flair/embeddings.py
--- a/flair/embeddings.py
+++ b/flair/embeddings.py
@@ -954,186 +954,109 @@ def __init__(
 
         cache_dir = Path("embeddings")
 
-        # multilingual forward (English, German, French, Italian, Dutch, Polish)
-        if model.lower() == "multi-forward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.4/lm-multi-forward-v0.1.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-        # multilingual backward  (English, German, French, Italian, Dutch, Polish)
-        elif model.lower() == "multi-backward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.4/lm-multi-backward-v0.1.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-
-        # multilingual forward fast (English, German, French, Italian, Dutch, Polish)
-        elif model.lower() == "multi-forward-fast":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.4/lm-multi-forward-fast-v0.1.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-        # multilingual backward fast (English, German, French, Italian, Dutch, Polish)
-        elif model.lower() == "multi-backward-fast":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.4/lm-multi-backward-fast-v0.1.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-
-        # news-english-forward
-        elif model.lower() == "news-forward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.4.1/big-news-forward--h2048-l1-d0.05-lr30-0.25-20/news-forward-0.4.1.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-
-        # news-english-backward
-        elif model.lower() == "news-backward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.4.1/big-news-backward--h2048-l1-d0.05-lr30-0.25-20/news-backward-0.4.1.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-
-        # news-english-forward
-        elif model.lower() == "news-forward-fast":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings/lm-news-english-forward-1024-v0.2rc.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-
-        # news-english-backward
-        elif model.lower() == "news-backward-fast":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings/lm-news-english-backward-1024-v0.2rc.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-
-        # mix-english-forward
-        elif model.lower() == "mix-forward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings/lm-mix-english-forward-v0.2rc.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-
-        # mix-english-backward
-        elif model.lower() == "mix-backward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings/lm-mix-english-backward-v0.2rc.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-
-        # mix-german-forward
-        elif model.lower() == "german-forward" or model.lower() == "de-forward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings/lm-mix-german-forward-v0.2rc.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-
-        # mix-german-backward
-        elif model.lower() == "german-backward" or model.lower() == "de-backward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings/lm-mix-german-backward-v0.2rc.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-
-        # common crawl Polish forward
-        elif model.lower() == "polish-forward" or model.lower() == "pl-forward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings/lm-polish-forward-v0.2.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-
-        # common crawl Polish backward
-        elif model.lower() == "polish-backward" or model.lower() == "pl-backward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings/lm-polish-backward-v0.2.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-
-        # Slovenian forward
-        elif model.lower() == "slovenian-forward" or model.lower() == "sl-forward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.3/lm-sl-large-forward-v0.1.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-        # Slovenian backward
-        elif model.lower() == "slovenian-backward" or model.lower() == "sl-backward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.3/lm-sl-large-backward-v0.1.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-
-        # Bulgarian forward
-        elif model.lower() == "bulgarian-forward" or model.lower() == "bg-forward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.3/lm-bg-small-forward-v0.1.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-        # Bulgarian backward
-        elif model.lower() == "bulgarian-backward" or model.lower() == "bg-backward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.3/lm-bg-small-backward-v0.1.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-
-        # Dutch forward
-        elif model.lower() == "dutch-forward" or model.lower() == "nl-forward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.4/lm-nl-large-forward-v0.1.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-        # Dutch backward
-        elif model.lower() == "dutch-backward" or model.lower() == "nl-backward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.4/lm-nl-large-backward-v0.1.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-
-        # Swedish forward
-        elif model.lower() == "swedish-forward" or model.lower() == "sv-forward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.4/lm-sv-large-forward-v0.1.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-        # Swedish backward
-        elif model.lower() == "swedish-backward" or model.lower() == "sv-backward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.4/lm-sv-large-backward-v0.1.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-
-        # French forward
-        elif model.lower() == "french-forward" or model.lower() == "fr-forward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings/lm-fr-charlm-forward.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-        # French backward
-        elif model.lower() == "french-backward" or model.lower() == "fr-backward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings/lm-fr-charlm-backward.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-
-        # Czech forward
-        elif model.lower() == "czech-forward" or model.lower() == "cs-forward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.4/lm-cs-large-forward-v0.1.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-        # Czech backward
-        elif model.lower() == "czech-backward" or model.lower() == "cs-backward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.4/lm-cs-large-backward-v0.1.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-
-        # Portuguese forward
-        elif model.lower() == "portuguese-forward" or model.lower() == "pt-forward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.4/lm-pt-forward.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-        # Portuguese backward
-        elif model.lower() == "portuguese-backward" or model.lower() == "pt-backward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.4/lm-pt-backward.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-
-        # Basque forward
-        elif model.lower() == "basque-forward" or model.lower() == "eu-forward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.4/lm-eu-large-forward-v0.1.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-        # Basque backward
-        elif model.lower() == "basque-backward" or model.lower() == "eu-backward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.4/lm-eu-large-backward-v0.1.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-
-        # Spanish forward fast
-        elif (
-            model.lower() == "spanish-forward-fast"
-            or model.lower() == "es-forward-fast"
-        ):
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.4/language_model_es_forward/lm-es-forward-fast.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-        # Spanish backward fast
-        elif (
-            model.lower() == "spanish-backward-fast"
-            or model.lower() == "es-backward-fast"
-        ):
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.4/language_model_es_backward/lm-es-backward-fast.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-
-        # Spanish forward
-        elif model.lower() == "spanish-forward" or model.lower() == "es-forward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.4/language_model_es_forward_long/lm-es-forward.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-        # Spanish backward
-        elif model.lower() == "spanish-backward" or model.lower() == "es-backward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.4/language_model_es_backward_long/lm-es-backward.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-
-        # Pubmed forward
-        elif model.lower() == "pubmed-forward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.4.1/pubmed-2015-fw-lm.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-        # Pubmed backward
-        elif model.lower() == "pubmed-backward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.4.1/pubmed-2015-bw-lm.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-
-        # Japanese forward
-        elif model.lower() == "japanese-forward" or model.lower() == "ja-forward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.4.1/lm__char-forward__ja-wikipedia-3GB/japanese-forward.pt"
-            model = cached_path(base_path, cache_dir=cache_dir)
-        # Japanese backward
-        elif model.lower() == "japanese-backward" or model.lower() == "ja-forward":
-            base_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/embeddings-v0.4.1/lm__char-backward__ja-wikipedia-3GB/japanese-backward.pt"
+        aws_path: str = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources"
+
+        self.PRETRAINED_MODEL_ARCHIVE_MAP = {
+            # multilingual models
+            "multi-forward": f"{aws_path}/embeddings-v0.4/lm-multi-forward-v0.1.pt",
+            "multi-backward": f"{aws_path}/embeddings-v0.4/lm-multi-backward-v0.1.pt",
+            "multi-forward-fast": f"{aws_path}/embeddings-v0.4/lm-multi-forward-fast-v0.1.pt",
+            "multi-backward-fast": f"{aws_path}/embeddings-v0.4/lm-multi-backward-fast-v0.1.pt",
+            # English models
+            "news-forward": f"{aws_path}/embeddings-v0.4.1/big-news-forward--h2048-l1-d0.05-lr30-0.25-20/news-forward-0.4.1.pt",
+            "news-backward": f"{aws_path}/embeddings-v0.4.1/big-news-backward--h2048-l1-d0.05-lr30-0.25-20/news-backward-0.4.1.pt",
+            "news-forward-fast": f"{aws_path}/embeddings/lm-news-english-forward-1024-v0.2rc.pt",
+            "news-backward-fast": f"{aws_path}/embeddings/lm-news-english-backward-1024-v0.2rc.pt",
+            "mix-forward": f"{aws_path}/embeddings/lm-mix-english-forward-v0.2rc.pt",
+            "mix-backward": f"{aws_path}/embeddings/lm-mix-english-backward-v0.2rc.pt",
+            # Arabic
+            "ar-forward": f"{aws_path}/embeddings-stefan-it/lm-ar-opus-large-forward-v0.1.pt",
+            "ar-backward": f"{aws_path}/embeddings-stefan-it/lm-ar-opus-large-backward-v0.1.pt",
+            # Bulgarian
+            "bg-forward-fast": f"{aws_path}/embeddings-v0.3/lm-bg-small-forward-v0.1.pt",
+            "bg-backward-fast": f"{aws_path}/embeddings-v0.3/lm-bg-small-backward-v0.1.pt",
+            "bg-forward": f"{aws_path}/embeddings-stefan-it/lm-bg-opus-large-forward-v0.1.pt",
+            "bg-backward": f"{aws_path}/embeddings-stefan-it/lm-bg-opus-large-backward-v0.1.pt",
+            # Czech
+            "cs-forward": f"{aws_path}/embeddings-stefan-it/lm-cs-opus-large-forward-v0.1.pt",
+            "cs-backward": f"{aws_path}/embeddings-stefan-it/lm-cs-opus-large-backward-v0.1.pt",
+            # Danish
+            "da-forward": f"{aws_path}/embeddings-stefan-it/lm-da-opus-large-forward-v0.1.pt",
+            "da-backward": f"{aws_path}/embeddings-stefan-it/lm-da-opus-large-backward-v0.1.pt",
+            # German
+            "de-forward": f"{aws_path}/embeddings/lm-mix-german-forward-v0.2rc.pt",
+            "de-backward": f"{aws_path}/embeddings/lm-mix-german-backward-v0.2rc.pt",
+            "de-historic-ha-forward": f"{aws_path}/embeddings-stefan-it/lm-historic-hamburger-anzeiger-forward-v0.1.pt",
+            "de-historic-ha-backward": f"{aws_path}/embeddings-stefan-it/lm-historic-hamburger-anzeiger-backward-v0.1.pt",
+            "de-historic-wz-forward": f"{aws_path}/embeddings-stefan-it/lm-historic-wiener-zeitung-forward-v0.1.pt",
+            "de-historic-wz-backward": f"{aws_path}/embeddings-stefan-it/lm-historic-wiener-zeitung-backward-v0.1.pt",
+            # Spanish
+            "es-forward": f"{aws_path}/embeddings-v0.4/language_model_es_forward_long/lm-es-forward.pt",
+            "es-backward": f"{aws_path}/embeddings-v0.4/language_model_es_backward_long/lm-es-backward.pt",
+            "es-forward-fast": f"{aws_path}/embeddings-v0.4/language_model_es_forward/lm-es-forward-fast.pt",
+            "es-backward-fast": f"{aws_path}/embeddings-v0.4/language_model_es_backward/lm-es-backward-fast.pt",
+            # Basque
+            "eu-forward": f"{aws_path}/embeddings-stefan-it/lm-eu-opus-large-forward-v0.1.pt",
+            "eu-backward": f"{aws_path}/embeddings-stefan-it/lm-eu-opus-large-backward-v0.1.pt",
+            # Farsi
+            "fa-forward": f"{aws_path}/embeddings-stefan-it/lm-fa-opus-large-forward-v0.1.pt",
+            "fa-backward": f"{aws_path}/embeddings-stefan-it/lm-fa-opus-large-backward-v0.1.pt",
+            # Finnish
+            "fi-forward": f"{aws_path}/embeddings-stefan-it/lm-fi-opus-large-forward-v0.1.pt",
+            "fi-backward": f"{aws_path}/embeddings-stefan-it/lm-fi-opus-large-backward-v0.1.pt",
+            # French
+            "fr-forward": f"{aws_path}/embeddings/lm-fr-charlm-forward.pt",
+            "fr-backward": f"{aws_path}/embeddings/lm-fr-charlm-backward.pt",
+            # Hebrew
+            "he-forward": f"{aws_path}/embeddings-stefan-it/lm-he-opus-large-forward-v0.1.pt",
+            "he-backward": f"{aws_path}/embeddings-stefan-it/lm-he-opus-large-backward-v0.1.pt",
+            # Hindi
+            "hi-forward": f"{aws_path}/embeddings-stefan-it/lm-hi-opus-large-forward-v0.1.pt",
+            "hi-backward": f"{aws_path}/embeddings-stefan-it/lm-hi-opus-large-backward-v0.1.pt",
+            # Croatian
+            "hr-forward": f"{aws_path}/embeddings-stefan-it/lm-hr-opus-large-forward-v0.1.pt",
+            "hr-backward": f"{aws_path}/embeddings-stefan-it/lm-hr-opus-large-backward-v0.1.pt",
+            # Indonesian
+            "id-forward": f"{aws_path}/embeddings-stefan-it/lm-id-opus-large-forward-v0.1.pt",
+            "id-backward": f"{aws_path}/embeddings-stefan-it/lm-id-opus-large-backward-v0.1.pt",
+            # Italian
+            "it-forward": f"{aws_path}/embeddings-stefan-it/lm-it-opus-large-forward-v0.1.pt",
+            "it-backward": f"{aws_path}/embeddings-stefan-it/lm-it-opus-large-backward-v0.1.pt",
+            # Japanese
+            "ja-forward": f"{aws_path}/embeddings-v0.4.1/lm__char-forward__ja-wikipedia-3GB/japanese-forward.pt",
+            "ja-backward": f"{aws_path}/embeddings-v0.4.1/lm__char-backward__ja-wikipedia-3GB/japanese-backward.pt",
+            # Dutch
+            "nl-forward": f"{aws_path}/embeddings-stefan-it/lm-nl-opus-large-forward-v0.1.pt",
+            "nl-backward": f"{aws_path}/embeddings-stefan-it/lm-nl-opus-large-backward-v0.1.pt",
+            # Norwegian
+            "no-forward": f"{aws_path}/embeddings-stefan-it/lm-no-opus-large-forward-v0.1.pt",
+            "no-backward": f"{aws_path}/embeddings-stefan-it/lm-no-opus-large-backward-v0.1.pt",
+            # Polish
+            "pl-forward": f"{aws_path}/embeddings/lm-polish-forward-v0.2.pt",
+            "pl-backward": f"{aws_path}/embeddings/lm-polish-backward-v0.2.pt",
+            # Portuguese
+            "pt-forward": f"{aws_path}/embeddings-v0.4/lm-pt-forward.pt",
+            "pt-backward": f"{aws_path}/embeddings-v0.4/lm-pt-backward.pt",
+            # Pubmed
+            "pubmed-forward": f"{aws_path}/embeddings-v0.4.1/pubmed-2015-fw-lm.pt",
+            "pubmed-backward": f"{aws_path}/embeddings-v0.4.1/pubmed-2015-bw-lm.pt",
+            # Slovenian
+            "sl-forward": f"{aws_path}/embeddings-stefan-it/lm-sl-opus-large-forward-v0.1.pt",
+            "sl-backward": f"{aws_path}/embeddings-stefan-it/lm-sl-opus-large-backward-v0.1.pt",
+            # Swedish
+            "sv-forward": f"{aws_path}/embeddings-stefan-it/lm-sv-opus-large-forward-v0.1.pt",
+            "sv-backward": f"{aws_path}/embeddings-stefan-it/lm-sv-opus-large-backward-v0.1.pt",
+        }
+
+        # load model if in pretrained model map
+        if model.lower() in self.PRETRAINED_MODEL_ARCHIVE_MAP:
+            base_path = self.PRETRAINED_MODEL_ARCHIVE_MAP[model.lower()]
+            model = cached_path(base_path, cache_dir=cache_dir)
+
+        elif replace_with_language_code(model) in self.PRETRAINED_MODEL_ARCHIVE_MAP:
+            base_path = self.PRETRAINED_MODEL_ARCHIVE_MAP[
+                replace_with_language_code(model)
+            ]
             model = cached_path(base_path, cache_dir=cache_dir)
 
         elif not Path(model).exists():
@@ -2367,7 +2290,7 @@ def _add_embeddings_internal(self, sentences: List[Sentence]):
                 if embedding.is_forward_lm:
                     sentence.set_embedding(
                         embedding.name,
-                        sentence[len(sentence)-1]._embeddings[embedding.name],
+                        sentence[len(sentence) - 1]._embeddings[embedding.name],
                     )
                 else:
                     sentence.set_embedding(
@@ -2430,3 +2353,29 @@ def embedding_length(self) -> int:
 
     def __str__(self):
         return self.name
+
+
+def replace_with_language_code(string: str):
+    string = string.replace("arabic-", "ar-")
+    string = string.replace("basque-", "eu-")
+    string = string.replace("bulgarian-", "bg-")
+    string = string.replace("croatian-", "hr-")
+    string = string.replace("czech-", "cs-")
+    string = string.replace("danish-", "da-")
+    string = string.replace("dutch-", "nl-")
+    string = string.replace("farsi-", "fa-")
+    string = string.replace("finnish-", "fi-")
+    string = string.replace("french-", "fr-")
+    string = string.replace("german-", "de-")
+    string = string.replace("hebrew-", "he-")
+    string = string.replace("hindi-", "hi-")
+    string = string.replace("indonesian-", "id-")
+    string = string.replace("italian-", "it-")
+    string = string.replace("japanese-", "ja-")
+    string = string.replace("norwegian-", "no")
+    string = string.replace("polish-", "pl-")
+    string = string.replace("portuguese-", "pt-")
+    string = string.replace("slovenian-", "sl-")
+    string = string.replace("spanish-", "es-")
+    string = string.replace("swedish-", "sv-")
+    return string


Test patch: