internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: flairNLP/flair

Issue:
print function for Dictionary class
Currently, the dictionary class only prints an object pointer:

```python
corpus = flair.datasets.UD_ENGLISH(in_memory=False)

tag_dictionary = corpus.make_tag_dictionary(tag_type='upos')
print(tag_dictionary)
```

This prints: 
```console
<flair.data.Dictionary object at 0x7f83187fcb50>
```

Better would be a printout that shows the number of items in dictionary and lists them. 


Discussion:

Resolving patch:
diff --git a/flair/data.py b/flair/data.py
--- a/flair/data.py
+++ b/flair/data.py
@@ -131,6 +131,10 @@ def load(cls, name: str):
 
         return Dictionary.load_from_file(name)
 
+    def __str__(self):
+        tags = ', '.join(self.get_item_for_index(i) for i in range(min(len(self), 30)))
+        return f"Dictionary with {len(self)} tags: {tags}"
+
 
 class Label:
     """


Test patch:
Repository: flairNLP/flair

Issue:
Embed the POS information when training NER model
Hi, the developers of Flair, I am really thankful that you guys make this very easily used library for neural NER task. 

Currently I am working on my language's NER task and I am wondering if I can include (or embed) the POS tag information when my NER model is trained, to increase the supervision of the context of the sentence (I am thinking the model could also learn the context of the NER tag based on the POS tag).

Thank you very much for your help!


Discussion:
Yes, by adding "tags" with POS to tokens and appending `OneHotEmbeddings(field='name_of_tag')`.
Should this be documented under [`resources/docs/embeddings`](https://github.com/flairNLP/flair/tree/4ce32c774b4dc5a8bfc0559441a1c8da4f06131d/resources/docs/embeddings)? And possibly also at [Tutorial 4: List of All Word Embeddings](https://github.com/flairNLP/flair/blob/4ce32c774b4dc5a8bfc0559441a1c8da4f06131d/resources/docs/TUTORIAL_4_ELMO_BERT_FLAIR_EMBEDDING.md)?

Resolving patch:
diff --git a/flair/embeddings.py b/flair/embeddings.py
--- a/flair/embeddings.py
+++ b/flair/embeddings.py
@@ -462,7 +462,7 @@ def extra_repr(self):
 
 
 class OneHotEmbeddings(TokenEmbeddings):
-    """One-hot encoded embeddings."""
+    """One-hot encoded embeddings. """
 
     def __init__(
         self,
@@ -471,7 +471,13 @@ def __init__(
         embedding_length: int = 300,
         min_freq: int = 3,
     ):
-
+        """
+        Initializes one-hot encoded word embeddings and a trainable embedding layer
+        :param corpus: you need to pass a Corpus in order to construct the vocabulary
+        :param field: by default, the 'text' of tokens is embedded, but you can also embed tags such as 'pos'
+        :param embedding_length: dimensionality of the trainable embedding layer
+        :param min_freq: minimum frequency of a word to become part of the vocabulary
+        """
         super().__init__()
         self.name = "one-hot"
         self.static_embeddings = False


Test patch:
Repository: flairNLP/flair

Issue:
Add Historical German Language Model
The research project @redewiedergabe has trained a language model on historical German fictional and non-fictional texts written between 1840 and 1920 with a total of 100,647,373 tokens, which they have offered to contribute to Flair.

They use it to tag different types of speech/thought/writing representation in German texts. More information on the DFG-project [here](http://redewiedergabe.de/), the tagger is available and described in detail [here](https://github.com/redewiedergabe/tagger). 

**Task**: Integrate the redewiedergabe language models into Flair. 


Discussion:

Resolving patch:
diff --git a/flair/embeddings.py b/flair/embeddings.py
--- a/flair/embeddings.py
+++ b/flair/embeddings.py
@@ -1852,6 +1852,7 @@ def __init__(self, model, fine_tune: bool = False, chars_per_chunk: int = 512):
         cache_dir = Path("embeddings")
 
         aws_path: str = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources"
+        hu_path: str = "https://flair.informatik.hu-berlin.de/resources"
 
         self.PRETRAINED_MODEL_ARCHIVE_MAP = {
             # multilingual models
@@ -1895,6 +1896,8 @@ def __init__(self, model, fine_tune: bool = False, chars_per_chunk: int = 512):
             "de-historic-ha-backward": f"{aws_path}/embeddings-stefan-it/lm-historic-hamburger-anzeiger-backward-v0.1.pt",
             "de-historic-wz-forward": f"{aws_path}/embeddings-stefan-it/lm-historic-wiener-zeitung-forward-v0.1.pt",
             "de-historic-wz-backward": f"{aws_path}/embeddings-stefan-it/lm-historic-wiener-zeitung-backward-v0.1.pt",
+            "de-historic-rw-forward": f"{hu_path}/embeddings/redewiedergabe_lm_forward.pt",
+            "de-historic-rw-backward": f"{hu_path}/embeddings/redewiedergabe_lm_backward.pt",
             # Spanish
             "es-forward": f"{aws_path}/embeddings-v0.4/language_model_es_forward_long/lm-es-forward.pt",
             "es-backward": f"{aws_path}/embeddings-v0.4/language_model_es_backward_long/lm-es-backward.pt",


Test patch:
Repository: flairNLP/flair

Issue:
Fine-grained POS taggers
Our current POS models for English and German use universal dependency tags. But for some use cases you might prefer fine-grained POS tags (see #1349). 

This issue tracks the addition of fine-grained models for English and German: 

- [ ] English POS model
- [x] Fast English POS model
- [x] German POS model 


Discussion:

Resolving patch:
diff --git a/flair/models/sequence_tagger_model.py b/flair/models/sequence_tagger_model.py
--- a/flair/models/sequence_tagger_model.py
+++ b/flair/models/sequence_tagger_model.py
@@ -866,9 +866,8 @@ def _fetch_model(model_name) -> str:
 
         model_map = {}
 
-        aws_resource_path_v04 = (
-            "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/models-v0.4"
-        )
+        aws_resource_path_v04 = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/models-v0.4"
+        hu_path: str = "https://nlp.informatik.hu-berlin.de/resources/models"
 
         model_map["ner"] = "/".join(
             [aws_resource_path_v04, "NER-conll03-english", "en-ner-conll03-v0.4.pt"]
@@ -921,7 +920,7 @@ def _fetch_model(model_name) -> str:
                 ]
             )
 
-        model_map["pos"] = "/".join(
+        model_map["upos"] = "/".join(
             [
                 aws_resource_path_v04,
                 "POS-ontonotes--h256-l1-b32-p3-0.5-%2Bglove%2Bnews-forward%2Bnews-backward-normal-locked0.5-word0.05--v0.4_0",
@@ -929,7 +928,15 @@ def _fetch_model(model_name) -> str:
             ]
         )
 
-        model_map["pos-fast"] = "/".join(
+        model_map["pos"] = "/".join(
+            [
+                hu_path,
+                "release-pos-0",
+                "en-pos-ontonotes-v0.5.pt",
+            ]
+        )
+
+        model_map["upos-fast"] = "/".join(
             [
                 aws_resource_path_v04,
                 "release-pos-fast-0",
@@ -937,6 +944,14 @@ def _fetch_model(model_name) -> str:
             ]
         )
 
+        model_map["pos-fast"] = "/".join(
+            [
+                hu_path,
+                "release-pos-fast-0",
+                "en-pos-ontonotes-fast-v0.5.pt",
+            ]
+        )
+
         for key in ["pos-multi", "multi-pos"]:
             model_map[key] = "/".join(
                 [
@@ -988,7 +1003,7 @@ def _fetch_model(model_name) -> str:
         )
 
         model_map["de-pos"] = "/".join(
-            [aws_resource_path_v04, "release-de-pos-0", "de-pos-ud-hdt-v0.4.pt"]
+            [hu_path, "release-de-pos-0", "de-pos-ud-hdt-v0.5.pt"]
         )
 
         model_map["de-pos-fine-grained"] = "/".join(


Test patch:
Repository: flairNLP/flair

Issue:
ClassificationCorpus not using all data provided
**Describe the bug**
ClassificationCorpus is not using all the data provided in 3 txt files (train.txt, dev.txt and test.txt). I suspect this to be an encoding issue, however Pandas is able to read in all the data with the same encoding type (utf-16) provided as parameter. 

**To Reproduce**
This [colab notebook](https://colab.research.google.com/drive/14tnyFZhnQHppb8fim8ZLgEwT5hLZx2TB).

**Expected behavior**
I expect the ClassificationCorpus object to use all data provided. The weird thing is, when I read the txt data as Pandas dataframes, all data is read. But when I read the txt data as ClassificationCorpus objects, some data is not used, as can be seen by the difference in total length (see notebook). 

**Environment (please complete the following information):**
 - Google Colab
 - Flair: installed from source


Discussion:
@NielsRogge thanks for reporting this and sharing the steps to reproduce! @marcelmmm can you take a look at this?
The problem is that in the data file the label and the text is separated by a tab and not a blank.
E.g. one line looks like this:  __label__1\t100st Prunus padus 60/90\n
with a \t between __label__1 and 100st. Now if we have a line where the text consists of only one word the line is not processed since there is no blanks at all in the line. So all the lines with one word are omitted. 

It should work if you replace the tabs with blank spaces.
Since this is not a sensible restriction I will add the case where tabs are used to the code.

Resolving patch:
diff --git a/flair/datasets/document_classification.py b/flair/datasets/document_classification.py
--- a/flair/datasets/document_classification.py
+++ b/flair/datasets/document_classification.py
@@ -174,7 +174,7 @@ def __init__(
             line = f.readline()
             position = 0
             while line:
-                if "__label__" not in line or " " not in line:
+                if "__label__" not in line or (" " not in line and "\t" not in line):
                     position = f.tell()
                     line = f.readline()
                     continue


Test patch:
Repository: flairNLP/flair

Issue:
use_dropout not getting saved in checkpoint ???
I think the use_dropout parameter does not saved as part of state_dict due to bug in
 def _get_state_dict(self) 

as a result of which if we set a dropout layer in sequence tagger, and resume checkpoint training, the behavior changes.




Discussion:
@sankaran45 thanks for spotting this! I'll fix in a PR!

Resolving patch:
diff --git a/flair/models/sequence_tagger_model.py b/flair/models/sequence_tagger_model.py
--- a/flair/models/sequence_tagger_model.py
+++ b/flair/models/sequence_tagger_model.py
@@ -238,6 +238,7 @@ def _get_state_dict(self):
             "use_crf": self.use_crf,
             "use_rnn": self.use_rnn,
             "rnn_layers": self.rnn_layers,
+            "use_dropout": self.use_dropout,
             "use_word_dropout": self.use_word_dropout,
             "use_locked_dropout": self.use_locked_dropout,
             "rnn_type": self.rnn_type,


Test patch:
Repository: flairNLP/flair

Issue:
Warnings in Python 3.8
**Describe the bug**

* Deprecation warnings are raised due to invalid escape sequences. This can be fixed by using raw strings or escaping the literals. pyupgrade also helps in automatic conversion : https://github.com/asottile/pyupgrade/
* Syntax warning due to comparison of literals using is.

**To Reproduce**

```
find . -iname '*.py' | grep -v example | xargs -P4 -I{} python3.8 -Wall -m py_compile {}
./flair/datasets/sequence_labeling.py:24: DeprecationWarning: invalid escape sequence \s
  column_delimiter: str = "\s+",
./flair/datasets/sequence_labeling.py:106: DeprecationWarning: invalid escape sequence \s
  column_delimiter: str = "\s+",
./flair/data_fetcher.py:573: DeprecationWarning: invalid escape sequence \s
  fields: List[str] = re.split("\s+", line)
./flair/trainers/language_model_trainer.py:146: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if no is 0:
./flair/trainers/language_model_trainer.py:148: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if no is 1:
```

**Environment (please complete the following information):**
 - OS [e.g. iOS, Linux]: Linux
 - Version [e.g. flair-0.3.2]: master branch


Discussion:

Resolving patch:
diff --git a/flair/data_fetcher.py b/flair/data_fetcher.py
--- a/flair/data_fetcher.py
+++ b/flair/data_fetcher.py
@@ -570,7 +570,7 @@ def read_column_data(
                 sentence: Sentence = Sentence()
 
             else:
-                fields: List[str] = re.split("\s+", line)
+                fields: List[str] = re.split(r"\s+", line)
                 token = Token(fields[text_column])
                 for column in column_name_map:
                     if len(fields) > column:
diff --git a/flair/datasets/sequence_labeling.py b/flair/datasets/sequence_labeling.py
--- a/flair/datasets/sequence_labeling.py
+++ b/flair/datasets/sequence_labeling.py
@@ -21,7 +21,7 @@ def __init__(
             test_file=None,
             dev_file=None,
             tag_to_bioes=None,
-            column_delimiter: str = "\s+",
+            column_delimiter: str = r"\s+",
             comment_symbol: str = None,
             encoding: str = "utf-8",
             document_separator_token: str = None,
@@ -103,7 +103,7 @@ def __init__(
             path_to_column_file: Union[str, Path],
             column_name_map: Dict[int, str],
             tag_to_bioes: str = None,
-            column_delimiter: str = "\s+",
+            column_delimiter: str = r"\s+",
             comment_symbol: str = None,
             in_memory: bool = True,
             document_separator_token: str = None,
diff --git a/flair/trainers/language_model_trainer.py b/flair/trainers/language_model_trainer.py
--- a/flair/trainers/language_model_trainer.py
+++ b/flair/trainers/language_model_trainer.py
@@ -143,9 +143,9 @@ def charsplit(
     @staticmethod
     def random_casechange(line: str) -> str:
         no = random.randint(0, 99)
-        if no is 0:
+        if no == 0:
             line = line.lower()
-        if no is 1:
+        if no == 1:
             line = line.upper()
         return line
 


Test patch:
Repository: flairNLP/flair

Issue:
is corpus object reusable across ModelTrainer instances ?
I have three checkpoint files generated from a training run that uses PooledFlair embedding. Say chk10.pt, chk20.pt, chk30.pt.

I finalize using the following code in a for loop to get the F1 predictions out:

trainer: ModelTrainer = ModelTrainer.load_checkpoint(chkfile, corpus)
trainer.train('.',  checkpoint = False, train_with_dev=True, max_epochs=epochs)

I set the epochs to the value at which this checkpoint got generated. So 10, 20, 30 etc. So typically it goes straight to creating the final model and emitting the predictions.

This works perfectly fine for the first time in the loop, after which the predictions are quite wrong. Now instead of doing it in the loop, if i simply do just once by restarting the process i get the values i expect. This behavior happens only with PooledFlairEmbedding. Same program runs just fine with ElmoEmbedding, BertEmbedding.

So my question is why is this the case ? Is it because i create the corpus object outside the for loop and keep reusing it across different ModelTrainer instances ? 

It happens quite regularly for me. If needed i can make a small program and share.



Discussion:
Could you create a minimal example script to reproduce?
Please run attached script. Three log files will get generated - initial training+evaluation, reload of checkpoint + evaluation repeated two times. 

This happens in other datasets also but only with PooledFlairEmbedding. Now, i understand some variability is expected but this is huge. Interestingly if i restart the process i get deterministic results.

I will appreciate either an explanation or preferably a fix as its causing lot of problems for me in reproducing my results.

[upload.zip](https://github.com/flairNLP/flair/files/4679815/upload.zip)


Resolving patch:
diff --git a/flair/nn.py b/flair/nn.py
--- a/flair/nn.py
+++ b/flair/nn.py
@@ -85,7 +85,7 @@ def load(cls, model: Union[str, Path]):
             # load_big_file is a workaround by https://github.com/highway11git to load models on some Mac/Windows setups
             # see https://github.com/zalandoresearch/flair/issues/351
             f = file_utils.load_big_file(str(model_file))
-            state = torch.load(f, map_location=flair.device)
+            state = torch.load(f, map_location='cpu')
 
         model = cls._init_model_with_state_dict(state)
 


Test patch:
Repository: flairNLP/flair

Issue:
Deserialization of tokenizer gets loaded from huggingface
**Describe the bug**
At the deserialization of a model, the tokenizer gets loaded from huggingface (we're using TransformerWordEmbeddings). This throws an error when for instance using SCIBERT, because it is not supported in huggingface.

**To Reproduce**
Train a sequence tagger model using scibert. Then according to the docs: tagger = SequenceTagger("model_using_scibert"), which throws an errors since it is looking the scibert tokenizer in the huggingface lib.

**Expected behavior**
The tokenizer should not get loaded from huggingface if it is not available.

**Environment (please complete the following information):**
 - macOS Catalina
 - Version flair 0.5



Discussion:

Resolving patch:
diff --git a/flair/embeddings/token.py b/flair/embeddings/token.py
--- a/flair/embeddings/token.py
+++ b/flair/embeddings/token.py
@@ -1134,7 +1134,10 @@ def __setstate__(self, d):
 
         # reload tokenizer to get around serialization issues
         model_name = self.name.split('transformer-word-')[-1]
-        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
+        try:
+            self.tokenizer = AutoTokenizer.from_pretrained(model_name)
+        except:
+            pass
 
 
 class FastTextEmbeddings(TokenEmbeddings):


Test patch:
Repository: flairNLP/flair

Issue:
predict(sentence, multi_class_prob=True) in TextClassification no longer works 
**Describe the bug**
predict(sentence, multi_class_prob=True) used to give probabilities for all labels, but now only give out one label and associated probability.  I have to revert to earlier commit to produce multiple class probabilities (pip install --upgrade git+https://github.com/flairNLP/flair.git@63aeabf9a18bdf53af3bcba5bd80f43ac717656e). 

**To Reproduce**
Steps to reproduce the behavior (e.g. which model did you train? what parameters did you use? etc.).

sentence = Sentence("Growth weakens as investment drops, consumers fade")

finetuned_classifier.predict(sentence,multi_class_prob=True)

print(sentence.labels) 

**Expected behavior**
it should return: [1 (0.0), -1 (1.0), 0 (0.0)] 
given the training data has 3 classes. 

**Screenshots**
now above steps returns only: [0 (0.0)]

**Environment (please complete the following information):**
 - OS [e.g. iOS, Linux]: Mac, Linux virtualenv on Google Colab
 - Version [e.g. flair-0.3.2]: tried both flair-0.5 installed via "pip install flair" and also "pip install --upgrade git+https://github.com/flairNLP/flair.git"




Discussion:

Resolving patch:
diff --git a/flair/models/text_classification_model.py b/flair/models/text_classification_model.py
--- a/flair/models/text_classification_model.py
+++ b/flair/models/text_classification_model.py
@@ -231,7 +231,7 @@ def predict(
 
                 for (sentence, labels) in zip(batch, predicted_labels):
                     for label in labels:
-                        if self.multi_label:
+                        if self.multi_label or multi_class_prob:
                             sentence.add_label(label_name, label.value, label.score)
                         else:
                             sentence.set_label(label_name, label.value, label.score)


Test patch:
diff --git a/tests/test_text_classifier.py b/tests/test_text_classifier.py
--- a/tests/test_text_classifier.py
+++ b/tests/test_text_classifier.py
@@ -123,6 +123,8 @@ def test_train_load_use_classifier_with_prob(results_base_path, tasks_base_path)
 
     model.predict(sentence, multi_class_prob=True)
 
+    assert len(sentence.labels) > 1
+
     for label in sentence.labels:
         assert label.value is not None
         assert 0.0 <= label.score <= 1.0
Repository: flairNLP/flair

Issue:
ELMoEmbeddings not working
Hello! I have the same bug trying to use ELMoEmbeddings, I guess the reason might be in the new release of allennlp library, am i right?
```UnboundLocalError: local variable 'allennlp' referenced before assignment```

_Originally posted by @Alea4jacta6est in https://github.com/flairNLP/flair/issues/597#issuecomment-652643892_


Discussion:
Hello @Alea4jacta6est you have to install `allennlp` by running `pip install allennlp`.
![image](https://user-images.githubusercontent.com/26580860/86453309-3db68480-bd26-11ea-8ff0-781d299f3b4f.png)
You can see at the screen that i just 
`Successfully installed allennlp-1.0.0 jsonnet-0.16.0 jsonpickle-1.4.1 overrides-3.0.0 tensorboardX-2.0 tokenizers-0.7.0 transformers-2.11.0`

Hmm just updated to the latest `allennlp` locally and it still works. Can you try running a python shell and doing `import allennlp` - does this work?
Yep, I have the same issue. Have the latest `allennlp` installed.
Ok I see the issue now. Looks like something has changed in 1.0 that breaks our code.

It also seems the latest `allennlp` is incompatible with `transformers` 3.0. They specify `transformers>=2.10,<2.11`as requirement.
It looks like support for the `ELMoEmbedder` class was removed in allennlp 1.0, see https://github.com/allenai/allennlp/issues/4384. 

They suggest installing an earlier version, i.e. `pip install allennlp==0.9.0`. 
@alanakbik I will update the install command with this fixed 0.9.0 version for `ELMoEmbeddings` in a PR now :)

Resolving patch:
diff --git a/flair/embeddings/token.py b/flair/embeddings/token.py
--- a/flair/embeddings/token.py
+++ b/flair/embeddings/token.py
@@ -1612,7 +1612,7 @@ def __init__(
             log.warning("-" * 100)
             log.warning('ATTENTION! The library "allennlp" is not installed!')
             log.warning(
-                'To use ELMoEmbeddings, please first install with "pip install allennlp"'
+                'To use ELMoEmbeddings, please first install with "pip install allennlp==0.9.0"'
             )
             log.warning("-" * 100)
             pass


Test patch:
Repository: flairNLP/flair

Issue:
Inconsistent tokenizer use cause bad predictions ...
**Describe the bug**
I have a CSV training/test files that i use CSVClassificationCorpus to load and then train etc. The evaluate that runs after training works fine. Then i manually load the CSV file and for each line, i call Sentence(...) and then pass it to predict function. This time the results are arbitrary and poor. 

I looked at it a bit, and it turned out that by default Sentence uses SpaceTokenizer (if no use_tokenizer parameter) is passed.

OTOH, CSVClassificationCorpus uses SegtokTokenizer by default ...

Leading to completely different results in the default case of not specifying these parameters.

So i fixed it by passing use_tokenize=SegtokTokenizer to my Sentence call before invoking predict

Quite counter-intutitive .. not necessarily a bug but posting in case some one else runs into same issue



Discussion:
Yes that's a good point. We have long considered making segtok the default tokenizer instead, but are unsure if this is the best way to go.

Resolving patch:
diff --git a/flair/data.py b/flair/data.py
--- a/flair/data.py
+++ b/flair/data.py
@@ -529,7 +529,7 @@ class Sentence(DataPoint):
     def __init__(
         self,
         text: str = None,
-        use_tokenizer: Union[bool, Tokenizer] = False,
+        use_tokenizer: Union[bool, Tokenizer] = True,
         language_code: str = None,
         start_position: int = None
     ):


Test patch:
diff --git a/tests/test_data.py b/tests/test_data.py
--- a/tests/test_data.py
+++ b/tests/test_data.py
@@ -53,7 +53,7 @@ def test_create_sentence_on_empty_string():
 
 
 def test_create_sentence_without_tokenizer():
-    sentence: Sentence = Sentence("I love Berlin.")
+    sentence: Sentence = Sentence("I love Berlin.", use_tokenizer=False)
 
     assert 3 == len(sentence.tokens)
     assert 0 == sentence.tokens[0].start_pos