internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: flairNLP/flair

Issue:
training.log not closed before next model
**Describe the bug**
Training multiple models in one script results in the `training.log` files of earlier models including the log output of all subsequent models.

**To Reproduce**
Do something like this (which might naturally happen during some sort of grid search):

```
# [create classifier and corpus somehow]
for outdir in ['a', 'b']:
    ModelTrainer(classifier, corpus).train(outdir)
```

Note that the contents of `b/training.log` are also appended to the end of `a/training.log`.

**Expected behavior**
Each training.log file should contain only the log output involved in training that model.

**Environment (please complete the following information):**
 - Ubuntu
 - Flair 0.4.1


Discussion:

Resolving patch:
diff --git a/flair/trainers/trainer.py b/flair/trainers/trainer.py
--- a/flair/trainers/trainer.py
+++ b/flair/trainers/trainer.py
@@ -68,14 +68,27 @@ def train(
         if eval_mini_batch_size is None:
             eval_mini_batch_size = mini_batch_size
 
-        log.info(f'Model training base path: "{base_path}"')
-
         # cast string to Path
         if type(base_path) is str:
             base_path = Path(base_path)
 
-        add_file_handler(log, base_path / "training.log")
+        log_handler = add_file_handler(log, base_path / "training.log")
 
+        log_line(log)
+        log.info(f'Model: "{self.model}"')
+        log_line(log)
+        log.info(f'Corpus: "{self.corpus}"')
+        log_line(log)
+        log.info("Parameters:")
+        log.info(f' - learning_rate: "{learning_rate}"')
+        log.info(f' - mini_batch_size: "{mini_batch_size}"')
+        log.info(f' - patience: "{patience}"')
+        log.info(f' - anneal_factor: "{anneal_factor}"')
+        log.info(f' - max_epochs: "{max_epochs}"')
+        log.info(f' - shuffle: "{train_with_dev}"')
+        log.info(f' - train_with_dev: "{train_with_dev}"')
+        log_line(log)
+        log.info(f'Model training base path: "{base_path}"')
         log_line(log)
         log.info(f"Evaluation method: {evaluation_metric.name}")
 
@@ -324,6 +337,8 @@ def train(
             final_score = 0
             log.info("Test data not provided setting final score to 0")
 
+        log.removeHandler(log_handler)
+
         return {
             "test_score": final_score,
             "dev_score_history": dev_score_history,
diff --git a/flair/training_utils.py b/flair/training_utils.py
--- a/flair/training_utils.py
+++ b/flair/training_utils.py
@@ -344,3 +344,4 @@ def add_file_handler(log, output_file):
     formatter = logging.Formatter("%(asctime)-15s %(message)s")
     fh.setFormatter(formatter)
     log.addHandler(fh)
+    return fh


Test patch:
Repository: flairNLP/flair

Issue:
Trouble reproducing your GermEval 2014 results
Hi. I can't reproduce your F1 score of 84.65 for GermEval 2014. I used everything out of the box:
```
corpus = datasets.GERMEVAL()
tagger = flair.models.SequenceTagger.load('de-ner-germeval')
sentences = corpus.test.sentences
result, loss = tagger.evaluate(sentences)
print(result.detailed_results)
```

But my results are these:
```
MICRO_AVG: acc 0.6475 - f1-score 0.7861
MACRO_AVG: acc 0.3776 - f1-score 0.4910583333333334
LOC        tp: 1449 - fp: 240 - fn: 257 - tn: 1449 - precision: 0.8579 - recall: 0.8494 - accuracy: 0.7446 - f1-score: 0.8536
LOCderiv   tp: 486 - fp: 98 - fn: 75 - tn: 486 - precision: 0.8322 - recall: 0.8663 - accuracy: 0.7375 - f1-score: 0.8489
LOCpart    tp: 48 - fp: 31 - fn: 61 - tn: 48 - precision: 0.6076 - recall: 0.4404 - accuracy: 0.3429 - f1-score: 0.5107
ORG        tp: 795 - fp: 322 - fn: 355 - tn: 795 - precision: 0.7117 - recall: 0.6913 - accuracy: 0.5401 - f1-score: 0.7014
ORGderiv   tp: 0 - fp: 0 - fn: 8 - tn: 0 - precision: 0.0000 - recall: 0.0000 - accuracy: 0.0000 - f1-score: 0.0000
ORGpart    tp: 109 - fp: 105 - fn: 63 - tn: 109 - precision: 0.5093 - recall: 0.6337 - accuracy: 0.3935 - f1-score: 0.5647
OTH        tp: 372 - fp: 144 - fn: 325 - tn: 372 - precision: 0.7209 - recall: 0.5337 - accuracy: 0.4423 - f1-score: 0.6133
OTHderiv   tp: 10 - fp: 5 - fn: 29 - tn: 10 - precision: 0.6667 - recall: 0.2564 - accuracy: 0.2273 - f1-score: 0.3704
OTHpart    tp: 2 - fp: 4 - fn: 40 - tn: 2 - precision: 0.3333 - recall: 0.0476 - accuracy: 0.0435 - f1-score: 0.0833
PER        tp: 1487 - fp: 222 - fn: 152 - tn: 1487 - precision: 0.8701 - recall: 0.9073 - accuracy: 0.7990 - f1-score: 0.8883
PERderiv   tp: 2 - fp: 1 - fn: 9 - tn: 2 - precision: 0.6667 - recall: 0.1818 - accuracy: 0.1667 - f1-score: 0.2857
PERpart    tp: 5 - fp: 9 - fn: 39 - tn: 5 - precision: 0.3571 - recall: 0.1136 - accuracy: 0.0943 - f1-score: 0.1724
```

Did you use some other F1-measure than micro-averaged? Did you use a different set of classes?
Thanks


Discussion:
Thanks for reporting this - I was able to reproduce your results. I'll take a closer look at what's happening here and report back!

Resolving patch:
diff --git a/flair/models/sequence_tagger_model.py b/flair/models/sequence_tagger_model.py
--- a/flair/models/sequence_tagger_model.py
+++ b/flair/models/sequence_tagger_model.py
@@ -840,11 +840,7 @@ def _fetch_model(model_name) -> str:
         )
 
         model_map["de-ner-germeval"] = "/".join(
-            [
-                aws_resource_path,
-                "NER-germeval--h256-l1-b32-%2Bde-fasttext%2Bgerman-forward%2Bgerman-backward--v0.2",
-                "de-ner-germeval-v0.3.pt",
-            ]
+            [aws_resource_path_v04, "NER-germeval", "de-ner-germeval-0.4.1.pt"]
         )
 
         model_map["fr-ner"] = "/".join(


Test patch:
Repository: flairNLP/flair

Issue:
Setting flair.device to a different GPU than default (0) is overwritten by ELMoEmbeddings when using them in DocumentPoolEmbedding
**Problem**
Specified GPU ('cuda:1') was ignored in `DocumentPoolEmbedding` since I'm using `ELMoEmbeddings` together with flair embeddings and in line 570 of `embeddings.py`, the device to use for elmo embeddings is set to 0 so after all, the embeddings were still done on 'cuda:0'

**Idea**
Make it possible to initialize elmo embeddings with a specific GPU. I adapted `embeddings.py` accordingly and now it's working fine. Happy to contribute with a PR in case the necessity is seen by others as well !


Discussion:
Hello @helmersl yes a PR would be great since other users have reported problems with this in the past as well!

Resolving patch:
diff --git a/flair/embeddings.py b/flair/embeddings.py
--- a/flair/embeddings.py
+++ b/flair/embeddings.py
@@ -567,7 +567,13 @@ def __init__(
         # put on Cuda if available
         from flair import device
 
-        cuda_device = 0 if str(device) != "cpu" else -1
+        if re.fullmatch(r'cuda:[0-9]+', str(device)):
+            cuda_device = int(str(device).split(':')[-1])
+        elif str(device) == "cpu":
+            cuda_device = -1
+        else:
+            cuda_device = 0
+
         self.ee = allennlp.commands.elmo.ElmoEmbedder(
             options_file=options_file, weight_file=weight_file, cuda_device=cuda_device
         )


Test patch:
Repository: flairNLP/flair

Issue:
the num_worker parameterLanguageModelTrainer

**Is your feature/enhancement request related to a problem? Please describe.**
the num_worker parameters in LanguageModelTrainer seems to be hard coded.

**Describe the solution you'd like**
it would be great to set it as a parameter of the LanguageModelTrainer class.

**Additional context**
I am currently fine tuning  FlairEmbeddings on a relatively small corpus ~30 M words which fits on memory (so I use a single split). 
For some reason the datasets seems to be preprocessed between each epoch. As a results training takes more time between each Epoch than the actual training time of a single epoch.  Any advice on this particular scenario would be helpful.

Btw great work with this package it is very helpful 👍 If you need any help, I would be happy to contribute. 


Discussion:
@ArmandGiraud you're right - I'll put in a PR to make this a parameter. In the meantime, you can still change this parameter by doing: 

```python
# train your language model
trainer = LanguageModelTrainer(language_model, corpus)

# set the num workers
trainer.num_workers = 10

# train away
trainer.train('resources/taggers/language_model',
              sequence_length=10,
              mini_batch_size=10,
              max_epochs=10)
```


Resolving patch:
diff --git a/flair/trainers/language_model_trainer.py b/flair/trainers/language_model_trainer.py
--- a/flair/trainers/language_model_trainer.py
+++ b/flair/trainers/language_model_trainer.py
@@ -236,7 +236,6 @@ def __init__(
 
         self.loss_function = torch.nn.CrossEntropyLoss()
         self.log_interval = 100
-        self.num_workers = 2
         self.epoch = epoch
         self.split = split
         self.loss = loss
@@ -254,6 +253,7 @@ def train(
         max_epochs: int = 1000,
         checkpoint: bool = False,
         grow_to_sequence_length: int = 0,
+        num_workers: int = 2,
         use_amp: bool = False,
         amp_opt_level: str = "O1",
         **kwargs,
@@ -306,7 +306,7 @@ def train(
                 )
 
             training_generator = DataLoader(
-                self.corpus.train, shuffle=False, num_workers=self.num_workers
+                self.corpus.train, shuffle=False, num_workers=num_workers
             )
 
             for epoch in range(self.epoch, max_epochs):
@@ -314,7 +314,7 @@ def train(
                 # Shuffle training files randomly after serially iterating through corpus one
                 if epoch > 0:
                     training_generator = DataLoader(
-                        self.corpus.train, shuffle=True, num_workers=self.num_workers
+                        self.corpus.train, shuffle=True, num_workers=num_workers
                     )
                     self.model.save_checkpoint(
                         base_path / f"epoch_{epoch}.pt",


Test patch:
Repository: flairNLP/flair

Issue:
UD_GERMAN_HDT corpus does not download
**Describe the bug**

When instantiating `UD_GERMAN_HDT()` corpus, the download of the train files fails because they have been renamed (and multiplied!).

Error is: 

```console
OSError: HEAD request failed for url https://raw.githubusercontent.com/UniversalDependencies/UD_German-HDT/dev/de_hdt-ud-train-a.conllu with status code 404.
```

**To Reproduce**

```python
from flair.datasets import UD_GERMAN_HDT

corpus = UD_GERMAN_HDT()
```


Discussion:

Resolving patch:
diff --git a/flair/datasets.py b/flair/datasets.py
--- a/flair/datasets.py
+++ b/flair/datasets.py
@@ -1426,17 +1426,20 @@ def __init__(self, base_path=None, in_memory: bool = False):
         )
         cached_path(f"{ud_path}/de_hdt-ud-dev.conllu", Path("datasets") / dataset_name)
         cached_path(f"{ud_path}/de_hdt-ud-test.conllu", Path("datasets") / dataset_name)
-        cached_path(
-            f"{ud_path}/de_hdt-ud-train-a.conllu",
-            Path("datasets") / dataset_name / "original",
-        )
-        cached_path(
-            f"{ud_path}/de_hdt-ud-train-b.conllu",
-            Path("datasets") / dataset_name / "original",
-        )
-        data_path = Path(flair.cache_root) / "datasets" / dataset_name
 
-        train_filenames = ["de_hdt-ud-train-a.conllu", "de_hdt-ud-train-b.conllu"]
+        train_filenames = [
+            "de_hdt-ud-train-a-1.conllu",
+            "de_hdt-ud-train-a-2.conllu",
+            "de_hdt-ud-train-b-1.conllu",
+            "de_hdt-ud-train-b-2.conllu",
+        ]
+
+        for train_file in train_filenames:
+            cached_path(
+                f"{ud_path}/{train_file}", Path("datasets") / dataset_name / "original"
+            )
+
+        data_path = Path(flair.cache_root) / "datasets" / dataset_name
 
         new_train_file: Path = data_path / "de_hdt-ud-train-all.conllu"
 


Test patch:
Repository: flairNLP/flair

Issue:
Using One Hot Embeddings fails on gpu
One Hot Embedding fails when printing embeddings, It works if I use cpu, but not with gpu. I think I found an issue with the use of torch.nn.Embedding(...) in embedddings.py, OneHotEmbeddings.__init__(...).

I have,

```python
import flair, torch

&#35; flair.device = torch.device('cpu')
torch.device('cuda:0')

from flair.embeddings import OneHotEmbeddings

word_embedding = OneHotEmbeddings(corpus)
sentence = corpus.train[0]
word_embedding.embed(sentence)

for token in sentence:
    print(token)
    print(token.embedding)
```

If I explicitly add cuda in the embedding layer everything works as expected, but fails otherwise. Thus,

```python
&#35; model architecture
self.embedding_layer = torch.nn.Embedding(
    len(self.vocab_dictionary), self.__embedding_length
)
if torch.cuda.is_available() and flair.device.type != 'cpu':
    self.embedding_layer.cuda()
```

Without the " fix" I get an error message:

```console
Traceback (most recent call last):
  File ".../examples/flaircorpus.py", line 291, in <module>
    word_embedding.embed(sentence)
  File .../github/flair/flair/embeddings.py", line 85, in embed
    self._add_embeddings_internal(sentences)
  File ".../github/flair/flair/embeddings.py", line 522, in _add_embeddings_internal
    embedded = self.embedding_layer.forward(one_hot_sentences)
  File ".../lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 117, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File ".../lib/python3.6/site-packages/torch/nn/functional.py", line 1506, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected object of backend CPU but got backend CUDA for argument #3 'index'

```


Discussion:
@carljohanrehn thanks for reporting this! I was able to reproduce and found its a bug in the constructor of the `OneHotEmbeddings` class. Will put in a PR to fix this!

Resolving patch:
diff --git a/flair/embeddings.py b/flair/embeddings.py
--- a/flair/embeddings.py
+++ b/flair/embeddings.py
@@ -498,6 +498,8 @@ def __init__(
         )
         torch.nn.init.xavier_uniform_(self.embedding_layer.weight)
 
+        self.to(flair.device)
+
     @property
     def embedding_length(self) -> int:
         return self.__embedding_length


Test patch:
Repository: flairNLP/flair

Issue:
Contribution for support for the Danish Language
Hi,

I have trained Flair word embeddings on Danish using Wikipedia and Europarl corpus, and by using this a Part of Speech Tagger trained on the [Danish Dependency Treebank](https://github.com/UniversalDependencies/UD_Danish-DDT/blob/master/README.md) corpus. In our project we also annotated the Danish Denendecy treebank corpus with NER tags (LOC, ORG, PER), and trained a NER tagger using Flair. Here is the link to our github [DaNLP](https://github.com/alexandrainst/danlp).

I think it could be cool to contribute the models to the Flair framework such that it supports Danish. Is this something you are interested in and how can it be done?

Kind Regards
Amalie Pauli 


Discussion:
Hello @AmaliePauli that would be great - we'd very much appreciate a contribution that adds support for Danish!

The easiest is if you send me the models and I re-upload them on our server and include them in the code. Otherwise, if you prefer to host the models on your server you could also do the PR yourself. Either way we'll make sure to reference DaNLP in the docs and code! 
Hi @AmaliePauli really great project 🤗

Have you also performed a comparison with the recently integrated Danish Flair Embeddings 🤔 I did some benchmarks for PoS tagging with the current Danish Flair Embeddings on the Universal Dependencies v1.2 (see [here](https://github.com/stefan-it/flair-pos-tagging)) :)
Hi @alanakbik , That sound great, thanks. Here is a link to the models https://alexandrainstituttet-my.sharepoint.com/:f:/g/personal/amalie_nyhagen_alexandra_dk/Eoi-TL7IW8FLvE62BiQ47pMBm0ixshMY7ktWy8FQy7TCKQ
We plan to continue working on the models and hope to improved them using more data in the future (especially ner), and we also integrate the flair models in our framework.
@stefan-it cool that you have run comparison on PoS tagging, I have not seen before that you also have trained it for Danish, and I see that your Danish PoS tagger gets 98.51 where I got   97,14 based on a single run. 

Thanks - I think the 'da-ner' and 'da-pos' files might be empty in the folder?
Sorry, the files should be properly uploaded now. Thanks:)  

Resolving patch:
diff --git a/flair/models/sequence_tagger_model.py b/flair/models/sequence_tagger_model.py
--- a/flair/models/sequence_tagger_model.py
+++ b/flair/models/sequence_tagger_model.py
@@ -948,6 +948,14 @@ def _fetch_model(model_name) -> str:
             ]
         )
 
+        model_map["da-pos"] = "/".join(
+            [aws_resource_path_v04, "POS-danish", "da-pos-v0.1.pt"]
+        )
+
+        model_map["da-ner"] = "/".join(
+            [aws_resource_path_v04, "NER-danish", "da-ner-v0.1.pt"]
+        )
+
         model_map["de-pos"] = "/".join(
             [aws_resource_path_v04, "release-de-pos-0", "de-pos-ud-hdt-v0.4.pt"]
         )


Test patch:
Repository: flairNLP/flair

Issue:
OneHotEmbeddings has error in implementation
**Describe the bug**
The implementation of OneHotEmbeddings for fields other than text has two issues:
* the value used in the counter is the instance of the class token, not the value of the token
* during the computation of embedding in `_add_embeddings_internal` the type of the field is not taken into account

**To Reproduce**
Use OneHotEmbedding with field other than `text` and check the contents of the dictionary, which is printed out. Should include all values with count >= 3, but only includes the `<unk>`.

**Expected behavior**
The dictionary should include all values with count >= 3. During computation, the value of the specific field, not the text field should be used.

**Environment (please complete the following information):**
 - OS [Linux]:
 - Version [flair-0.4.3]:

**Additional context**
I will send a PR with the fix shortly.



Discussion:
@apohllo thanks for reporting this - look forward to the PR! :) 

Resolving patch:
diff --git a/flair/embeddings.py b/flair/embeddings.py
--- a/flair/embeddings.py
+++ b/flair/embeddings.py
@@ -449,7 +449,6 @@ def __str__(self):
     def extra_repr(self):
         return f"'{self.embeddings}'"
 
-
 class OneHotEmbeddings(TokenEmbeddings):
     """One-hot encoded embeddings."""
 
@@ -465,6 +464,7 @@ def __init__(
         self.name = "one-hot"
         self.static_embeddings = False
         self.min_freq = min_freq
+        self.field = field
 
         tokens = list(map((lambda s: s.tokens), corpus.train))
         tokens = [token for sublist in tokens for token in sublist]
@@ -473,7 +473,7 @@ def __init__(
             most_common = Counter(list(map((lambda t: t.text), tokens))).most_common()
         else:
             most_common = Counter(
-                list(map((lambda t: t.get_tag(field)), tokens))
+                list(map((lambda t: t.get_tag(field).value), tokens))
             ).most_common()
 
         tokens = []
@@ -508,9 +508,15 @@ def _add_embeddings_internal(self, sentences: List[Sentence]) -> List[Sentence]:
 
         one_hot_sentences = []
         for i, sentence in enumerate(sentences):
-            context_idxs = [
-                self.vocab_dictionary.get_idx_for_item(t.text) for t in sentence.tokens
-            ]
+
+            if self.field == "text":
+                context_idxs = [
+                    self.vocab_dictionary.get_idx_for_item(t.text) for t in sentence.tokens
+                ]
+            else:
+                context_idxs = [
+                    self.vocab_dictionary.get_idx_for_item(t.get_tag(self.field).value) for t in sentence.tokens
+                ]
 
             one_hot_sentences.extend(context_idxs)
 


Test patch:
Repository: flairNLP/flair

Issue:
Remove leftover print
**Describe the bug**
When trying to use muse embeddings, for each sentence the language code is printed.

**To Reproduce**
Please have a look at: https://github.com/zalandoresearch/flair/blob/31cc245f45b38dbbc4ddfc0214a2761a45b2b3a9/flair/embeddings.py#L581

**Expected behavior**
Train without printing language code for each sentence

**Environment (please complete the following information):**
 - OS [e.g. iOS, Linux]: Linux
 - Version [e.g. flair-0.3.2]: flair-0.4.3




Discussion:

Resolving patch:
diff --git a/flair/embeddings.py b/flair/embeddings.py
--- a/flair/embeddings.py
+++ b/flair/embeddings.py
@@ -581,7 +581,6 @@ def _add_embeddings_internal(self, sentences: List[Sentence]) -> List[Sentence]:
         for i, sentence in enumerate(sentences):
 
             language_code = sentence.get_language_code()
-            print(language_code)
             supported = [
                 "en",
                 "de",
diff --git a/flair/models/language_model.py b/flair/models/language_model.py
--- a/flair/models/language_model.py
+++ b/flair/models/language_model.py
@@ -354,8 +354,6 @@ def generate_text(
             if not self.is_forward_lm:
                 text = text[::-1]
 
-            text = text.encode("utf-8")
-
             return text, log_prob
 
     def calculate_perplexity(self, text: str) -> float:
diff --git a/flair/trainers/language_model_trainer.py b/flair/trainers/language_model_trainer.py
--- a/flair/trainers/language_model_trainer.py
+++ b/flair/trainers/language_model_trainer.py
@@ -278,6 +278,13 @@ def train(
 
         val_data = self._batchify(self.corpus.valid, mini_batch_size)
 
+        # error message if the validation dataset is too small
+        if val_data.size(0) == 1:
+            raise RuntimeError(
+                f"ERROR: Your validation dataset is too small. For your mini_batch_size, the data needs to "
+                f"consist of at least {mini_batch_size * 2} characters!"
+            )
+
         base_path.mkdir(parents=True, exist_ok=True)
         loss_txt = base_path / "loss.txt"
         savefile = base_path / "best-lm.pt"


Test patch:
Repository: flairNLP/flair

Issue:
Plotter throws error if test loss is not monitored
The Plotter method to plot training curves currently throws an error (
`ValueError: 'TEST_LOSS' is not in list`) if test loss is not monitored during training. 

To reproduce:

```python
trainer: ModelTrainer = ModelTrainer(tagger, corpus)

trainer.train(
    "resources/taggers/example-ner",
    max_epochs=1,
)

plotter = Plotter()
plotter.plot_training_curves("resources/taggers/example-ner/loss.tsv")
```





Discussion:

Resolving patch:
diff --git a/flair/visual/training_curves.py b/flair/visual/training_curves.py
--- a/flair/visual/training_curves.py
+++ b/flair/visual/training_curves.py
@@ -54,7 +54,7 @@ def _extract_evaluation_data(file_name: Path, score: str = "F1") -> dict:
                 row.index(f"TRAIN_{score}") if f"TRAIN_{score}" in row else None
             )
             DEV_SCORE = row.index(f"DEV_{score}") if f"DEV_{score}" in row else None
-            TEST_SCORE = row.index(f"TEST_{score}")
+            TEST_SCORE = row.index(f"TEST_{score}") if f"TEST_{score}" in row else None
 
             # then get all relevant values from the tsv
             for row in tsvin:
@@ -69,8 +69,9 @@ def _extract_evaluation_data(file_name: Path, score: str = "F1") -> dict:
                     if row[DEV_SCORE] != "_":
                         training_curves["dev"]["score"].append(float(row[DEV_SCORE]))
 
-                if row[TEST_SCORE] != "_":
-                    training_curves["test"]["score"].append(float(row[TEST_SCORE]))
+                if TEST_SCORE is not None:
+                    if row[TEST_SCORE] != "_":
+                        training_curves["test"]["score"].append(float(row[TEST_SCORE]))
 
         return training_curves
 


Test patch: