internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: flairNLP/flair

Issue:
Macro/Micro F scores in the Sequence tagger model
Hello,

In the Sequence tagger model,  (./flair/models/sequence_tagger_model.py), when the `evaluate` function computes the scores, it does the following:

line 576: `macro_f_score = round(metrics.fbeta_score(y_true, y_pred, beta=self.beta, average='micro'), 4)`
line 577: `micro_f_score = round(metrics.fbeta_score(y_true, y_pred, beta=self.beta, average='macro'), 4)`

Why is macro f score taking parameter `average=micro` and vice-versa?

Thank you in advance for your answer! 


Discussion:
@yusiZou thanks for spotting this! The variable names are flipped, but the printout is also flipped: 

```python
        detailed_result = (
                "\nResults:"
                f"\n- F-score (micro) {macro_f_score}"
                f"\n- F-score (macro) {micro_f_score}"
                f"\n- Accuracy {accuracy_score}"
                '\n\nBy class:\n' + classification_report
        )
```

So the two errors cancel each other out and the final printout is correct, but we will correct the variable names!

Resolving patch:
diff --git a/flair/models/sequence_tagger_model.py b/flair/models/sequence_tagger_model.py
--- a/flair/models/sequence_tagger_model.py
+++ b/flair/models/sequence_tagger_model.py
@@ -573,14 +573,14 @@ def evaluate(
                                                               zero_division=1)
 
         # get scores
-        macro_f_score = round(metrics.fbeta_score(y_true, y_pred, beta=self.beta, average='micro'), 4)
-        micro_f_score = round(metrics.fbeta_score(y_true, y_pred, beta=self.beta, average='macro'), 4)
+        micro_f_score = round(metrics.fbeta_score(y_true, y_pred, beta=self.beta, average='micro'), 4)
+        macro_f_score = round(metrics.fbeta_score(y_true, y_pred, beta=self.beta, average='macro'), 4)
         accuracy_score = round(metrics.accuracy_score(y_true, y_pred), 4)
 
         detailed_result = (
                 "\nResults:"
-                f"\n- F-score (micro) {macro_f_score}"
-                f"\n- F-score (macro) {micro_f_score}"
+                f"\n- F-score (micro) {micro_f_score}"
+                f"\n- F-score (macro) {macro_f_score}"
                 f"\n- Accuracy {accuracy_score}"
                 '\n\nBy class:\n' + classification_report
         )
@@ -590,7 +590,7 @@ def evaluate(
         log_line = f"\t{accuracy_score}"
 
         result = Result(
-            main_score=macro_f_score,
+            main_score=micro_f_score,
             log_line=log_line,
             log_header=log_header,
             detailed_results=detailed_result,


Test patch:
Repository: flairNLP/flair

Issue:
NER column corpus with custom delimiter doesn't work properly
**Describe the bug**
When using custom column delimiter for CoNLL format, the newline character is taken as a part of a tag.

**To Reproduce**
Create a CoNLL file, load setting column_delimiter, check loaded tags. Tags are 'O\n', 'B-something\n' and 'I-something\n'

**Expected behavior**
Tags should be 'O', 'B-something' or 'I-something'.



Discussion:
The problem is that reading the file is done using readline() which also reads the trailing newline. Splitting on default delimiter also splits an empty column after that newline. That's why the issue doesn't come up when using default delimiter. My current workaround is to add the delimiter after the NER tag. The correct solution is to trim NER tags when creating a corpus.

Resolving patch:
diff --git a/flair/datasets/sequence_labeling.py b/flair/datasets/sequence_labeling.py
--- a/flair/datasets/sequence_labeling.py
+++ b/flair/datasets/sequence_labeling.py
@@ -204,7 +204,7 @@ def __init__(
             self.total_sentence_count += 1
 
     def _parse_token(self, line: str) -> Token:
-        fields: List[str] = re.split(self.column_delimiter, line)
+        fields: List[str] = re.split(self.column_delimiter, line.rstrip())
         token = Token(fields[self.text_column])
         for column in self.column_name_map:
             if len(fields) > column:


Test patch:
Repository: flairNLP/flair

Issue:
AttributeError using de-offensive-language 
I tried to use the TextClassifier with '_de-offensive-language_':

```
classifier = TextClassifier.load('de-offensive-language')

sentence = Sentence('Das gefällt mir nicht.')

# predict has a Bug
classifier.predict(sentence)

```
I get the following Error:en-sentiment
```
....
  File "<...>/python3.7/site-packages/flair/embeddings.py", line 518, in _add_embeddings_internal
    if self.field == "text":
  File "<...>/python3.7/site-packages/torch/nn/modules/module.py", line 585, in __getattr__
    type(self).__name__, name))
AttributeError: 'OneHotEmbeddings' object has no attribute 'field'
```
Any ideas how to fix this? With _en-sentiment_ it works fine.

Thanks for your help.



Discussion:
The following fix helped:
`# predict has a Bug
classifier.document_embeddings.embeddings._modules['list_embedding_0'].field = "text"
classifier.predict(sentence)`

It seems that in flair 0.4.4 _OneHotEmbeddings_ have an attribute _field_ which is missing in the pretrained model. 
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

I can confirm that the bug still exists in flair 0.6.0 - luckily the fix still works, too

Resolving patch:
diff --git a/flair/models/text_classification_model.py b/flair/models/text_classification_model.py
--- a/flair/models/text_classification_model.py
+++ b/flair/models/text_classification_model.py
@@ -462,7 +462,7 @@ def _fetch_model(model_name) -> str:
         hu_path: str = "https://nlp.informatik.hu-berlin.de/resources/models"
 
         model_map["de-offensive-language"] = "/".join(
-            [hu_path, "de-offensive-language", "germ-eval-2018-task-1-v0.4.pt"]
+            [hu_path, "de-offensive-language", "germ-eval-2018-task-1-v0.5.pt"]
         )
 
         # English sentiment models


Test patch:
Repository: flairNLP/flair

Issue:
Prepare 0.6.1 release
Its time to do another release. Mostly to push recent bug fixed to pip, including the problems caused by moving the models from our original server to our university server.



Discussion:

Resolving patch:
diff --git a/flair/__init__.py b/flair/__init__.py
--- a/flair/__init__.py
+++ b/flair/__init__.py
@@ -24,7 +24,7 @@
 
 import logging.config
 
-__version__ = "0.6.0.post1"
+__version__ = "0.6.1"
 
 logging.config.dictConfig(
     {
diff --git a/flair/datasets/treebanks.py b/flair/datasets/treebanks.py
--- a/flair/datasets/treebanks.py
+++ b/flair/datasets/treebanks.py
@@ -1070,7 +1070,8 @@ def __init__(self, base_path: Union[str, Path] = None, in_memory: bool = True):
         )
 
         super(UD_BASQUE, self).__init__(data_folder, in_memory=in_memory)
-        
+
+
 class UD_CHINESE_KYOTO(UniversalDependenciesCorpus):
     def __init__(self, base_path: Union[str, Path] = None, in_memory: bool = True):
 
@@ -1095,4 +1096,4 @@ def __init__(self, base_path: Union[str, Path] = None, in_memory: bool = True):
             f"{web_path}/lzh_kyoto-ud-train.conllu", Path("datasets") / dataset_name
         )
 
-        super(UD_CHINESE_KYOTO, self).__init__(data_folder, in_memory=in_memory)        
+        super(UD_CHINESE_KYOTO, self).__init__(data_folder, in_memory=in_memory)
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -5,7 +5,7 @@
 
 setup(
     name="flair",
-    version="0.6.0.post1",
+    version="0.6.1",
     description="A very simple framework for state-of-the-art NLP",
     long_description=open("README.md", encoding="utf-8").read(),
     long_description_content_type="text/markdown",


Test patch:
Repository: flairNLP/flair

Issue:
Added support for the Livvi dataset from UD
This PR concludes my homework assignment.
It is not recommended to accept the PR at this time, since the data source is currently in the dev branch in my own rep (PR to UD pending) and I'm not qualified to determine if my changes have any linguistic implications.

My changes to the UD dataset:
- Swap the training and test dataset since the old test dataset was much larger than the training dataset. 
- Remove part of the (new) training dataset to create a dev dataset. (Ratios might need adjustment.)

In flair/flair/datasets/document_classification.py Line 48 and flair/flair/datasets/treebanks.py  Line 34 it is stated:
":param dev_file: the name of the dev file, if None, dev data is sampled from train."
This only holds true if the dataset is loaded manually. 
When automatically loaded from UD Line 67 in flair/flair/datasets/treebanks.py prevents this. since "assert path_to_conll_file.exists()" is called for all "path_to_conll_file", "None" or "str".


Discussion:
@hebecked thanks for adding this! As discussed [here](https://github.com/UniversalDependencies/UD_Livvi-KKPP/issues/2) can you use the original splits? I.e. the large test split and the small training split? 

Since there is no dev split, you can init the object only using a test and training split. The object `UD_CHINESE_KYOTO` is an example of a UD treebank initialized only with test and train.

Resolving patch:
diff --git a/flair/datasets/treebanks.py b/flair/datasets/treebanks.py
--- a/flair/datasets/treebanks.py
+++ b/flair/datasets/treebanks.py
@@ -44,10 +44,10 @@ def __init__(
         train = UniversalDependenciesDataset(train_file, in_memory=in_memory)
 
         # get test data
-        test = UniversalDependenciesDataset(test_file, in_memory=in_memory)
+        test = UniversalDependenciesDataset(test_file, in_memory=in_memory) if test_file is not None else None
 
         # get dev data
-        dev = UniversalDependenciesDataset(dev_file, in_memory=in_memory)
+        dev = UniversalDependenciesDataset(dev_file, in_memory=in_memory) if dev_file is not None else None
 
         super(UniversalDependenciesCorpus, self).__init__(
             train, dev, test, name=str(data_folder)


Test patch:
Repository: flairNLP/flair

Issue:
How to use pretokenized sequences?
Hello, I want to use pretokenized sequences (I **don't** want to use a different tokenizer, e.g. [docs](https://github.com/flairNLP/flair/blob/master/resources/docs/TUTORIAL_1_BASICS.md#using-a-different-tokenizer)).

Made up example:

```py
pretokenized_sequence = ['The', 'grass', 'is', 'green', '.']
my_sentence_object = Sentence(pretokenized_sequence, is_pretokenized=True)
```

Is it possible to achieve something like this? 



Discussion:
# Proposal
https://github.com/flairNLP/flair/blob/master/flair/data.py#L570

```py
if text is not None:
    text = self._restore_windows_1252_characters(text)
    [self.add_token(token) for token in tokenizer.tokenize(text)]
elif is_pretokenized  and isinstance(text, collections.abc.Iterable):
    [self.add_token(token) for token in text]
    # or [self.add_token(self._restore_windows_1252_characters(token)) for token in text]
```

I can make a PR if you want. 

Resolving patch:
diff --git a/flair/data.py b/flair/data.py
--- a/flair/data.py
+++ b/flair/data.py
@@ -525,14 +525,14 @@ class Sentence(DataPoint):
 
     def __init__(
         self,
-        text: str = None,
+        text: Union[str, List[str]] = None,
         use_tokenizer: Union[bool, Tokenizer] = True,
         language_code: str = None,
         start_position: int = None
     ):
         """
         Class to hold all meta related to a text (tokens, predictions, language code, ...)
-        :param text: original string
+        :param text: original string (sentence), or a list of string tokens (words)
         :param use_tokenizer: a custom tokenizer (default is :class:`SpaceTokenizer`)
             more advanced options are :class:`SegTokTokenizer` to use segtok or :class:`SpacyTokenizer`
             to use Spacy library if available). Check the implementations of abstract class Tokenizer or
@@ -568,8 +568,12 @@ def __init__(
 
         # if text is passed, instantiate sentence with tokens (words)
         if text is not None:
-            text = self._restore_windows_1252_characters(text)
-            [self.add_token(token) for token in tokenizer.tokenize(text)]
+            if isinstance(text, (list, tuple)):
+                [self.add_token(self._restore_windows_1252_characters(token))
+                 for token in text]
+            else:
+                text = self._restore_windows_1252_characters(text)
+                [self.add_token(token) for token in tokenizer.tokenize(text)]
 
         # log a warning if the dataset is empty
         if text == "":


Test patch:
diff --git a/tests/test_data.py b/tests/test_data.py
--- a/tests/test_data.py
+++ b/tests/test_data.py
@@ -866,3 +866,10 @@ def test_sentence_to_dict():
     assert "Facebook, Inc." == dict["entities"][0]["text"]
     assert "Google" == dict["entities"][1]["text"]
     assert 0 == len(dict["labels"])
+
+
+def test_pretokenized():
+    pretoks = ['The', 'grass', 'is', 'green', '.']
+    sent = Sentence(pretoks)
+    for i, token in enumerate(sent):
+        assert token.text == pretoks[i]
Repository: flairNLP/flair

Issue:
Do not shuffle on first epoch
Normally, we shuffle sentences at each epoch during training in the ModelTrainer class. However, in some cases it makes sense to see sentences in their natural order during the first epoch, and shuffle only from the second epoch onward. 

This PR monitors the progress of adding this feature to Flair.


Discussion:

Resolving patch:
diff --git a/flair/trainers/trainer.py b/flair/trainers/trainer.py
--- a/flair/trainers/trainer.py
+++ b/flair/trainers/trainer.py
@@ -342,7 +342,7 @@ def train(
                 batch_loader = DataLoader(
                     train_data,
                     batch_size=mini_batch_size,
-                    shuffle=shuffle,
+                    shuffle=shuffle if self.epoch > 1 else False, # never shuffle the first epoch
                     num_workers=num_workers,
                     sampler=sampler,
                 )
@@ -359,6 +359,7 @@ def train(
                 # process mini-batches
                 batch_time = 0
                 for batch_no, batch in enumerate(batch_loader):
+
                     start_time = time.time()
 
                     # zero the gradients on the model and optimizer


Test patch:
Repository: flairNLP/flair

Issue:
Update TARS model for Flair 0.8 release
Flair 0.8 includes some improvements to the TARS model. This issue tracks the progress of updating the tars-base model for the new release.


Discussion:

Resolving patch:
diff --git a/flair/models/text_classification_model.py b/flair/models/text_classification_model.py
--- a/flair/models/text_classification_model.py
+++ b/flair/models/text_classification_model.py
@@ -939,7 +939,7 @@ def _fetch_model(model_name) -> str:
         model_map = {}
         hu_path: str = "https://nlp.informatik.hu-berlin.de/resources/models"
 
-        model_map["tars-base"] = "/".join([hu_path, "tars-base", "tars-base-t4.pt"])
+        model_map["tars-base"] = "/".join([hu_path, "tars-base", "tars-base-v8.pt"])
 
         cache_dir = Path("models")
         if model_name in model_map:


Test patch:
Repository: flairNLP/flair

Issue:
Add FLERT models to HF hub
This issue tracks the progress of adding the new FLERT models to the HF hub.

- [x] NER English large
- [x] NER German large
- [x] NER Dutch large
- [x] NER Spanish large
- [x] NER English Ontonotes large 


Discussion:

Resolving patch:
diff --git a/flair/models/sequence_tagger_model.py b/flair/models/sequence_tagger_model.py
--- a/flair/models/sequence_tagger_model.py
+++ b/flair/models/sequence_tagger_model.py
@@ -1010,6 +1010,12 @@ def _fetch_model(model_name) -> str:
             "ner-fast": "flair/ner-english-fast",
             "ner-ontonotes": "flair/ner-english-ontonotes",
             "ner-ontonotes-fast": "flair/ner-english-ontonotes-fast",
+            # Large NER models,
+            "ner-large": "flair/ner-english-large",
+            "ner-ontonotes-large": "flair/ner-english-ontonotes-large",
+            "de-ner-large": "flair/ner-german-large",
+            "nl-ner-large": "flair/ner-dutch-large",
+            "es-ner-large": "flair/ner-spanish-large",
             # Multilingual NER models
             "ner-multi": "flair/ner-multi",
             "multi-ner": "flair/ner-multi",


Test patch:
Repository: flairNLP/flair

Issue:
Release Flair 0.8
Time finally for another release! This issue tracks the progress of releasing 0.8.


Discussion:

Resolving patch:
diff --git a/flair/__init__.py b/flair/__init__.py
--- a/flair/__init__.py
+++ b/flair/__init__.py
@@ -25,7 +25,7 @@
 
 import logging.config
 
-__version__ = "0.7"
+__version__ = "0.8"
 
 logging.config.dictConfig(
     {
diff --git a/flair/datasets/__init__.py b/flair/datasets/__init__.py
--- a/flair/datasets/__init__.py
+++ b/flair/datasets/__init__.py
@@ -10,6 +10,7 @@
 from .sequence_labeling import ANER_CORP
 from .sequence_labeling import BIOFID
 from .sequence_labeling import BIOSCOPE
+from .sequence_labeling import BUSINESS_HUN
 from .sequence_labeling import CONLL_03
 from .sequence_labeling import CONLL_03_GERMAN
 from .sequence_labeling import CONLL_03_DUTCH
@@ -55,7 +56,6 @@
 from .sequence_labeling import WSD_UFSAC
 from .sequence_labeling import WNUT_2020_NER
 from .sequence_labeling import XTREME
-from .sequence_labeling import BUSINESS_HUN
 
 # Expose all document classification datasets
 from .document_classification import ClassificationCorpus
@@ -63,6 +63,9 @@
 from .document_classification import CSVClassificationCorpus
 from .document_classification import CSVClassificationDataset
 from .document_classification import AMAZON_REVIEWS
+from .document_classification import COMMUNICATIVE_FUNCTIONS
+from .document_classification import GERMEVAL_2018_OFFENSIVE_LANGUAGE
+from .document_classification import GO_EMOTIONS
 from .document_classification import IMDB
 from .document_classification import NEWSGROUPS
 from .document_classification import SENTIMENT_140
@@ -74,13 +77,10 @@
 from .document_classification import SENTEVAL_SST_GRANULAR
 from .document_classification import TREC_50
 from .document_classification import TREC_6
-from .document_classification import COMMUNICATIVE_FUNCTIONS
 from .document_classification import WASSA_ANGER
 from .document_classification import WASSA_FEAR
 from .document_classification import WASSA_JOY
 from .document_classification import WASSA_SADNESS
-from .document_classification import GO_EMOTIONS
-from .document_classification import GERMEVAL_2018_OFFENSIVE_LANGUAGE
 
 # Expose all treebanks
 from .treebanks import UniversalDependenciesCorpus
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -5,7 +5,7 @@
 
 setup(
     name="flair",
-    version="0.7",
+    version="0.8",
     description="A very simple framework for state-of-the-art NLP",
     long_description=open("README.md", encoding="utf-8").read(),
     long_description_content_type="text/markdown",


Test patch:
Repository: flairNLP/flair

Issue:
Close log_handler after training is complete.
**Describe the bug**
We are removing the log_handler [here](https://github.com/flairNLP/flair/blob/master/flair/trainers/trainer.py#L633) but not closing the handler leading to  `ResourceWarning: unclosed file <_io.BufferedReader name='training.log`. Hence we are not able to programatically access the training.log (For ex - Unable to upload the file to s3 using botocore)

**To Reproduce**
Training just any model leads to this Warning. 

**Expected behavior**
The `log_handler` to be closed before training exits

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Environment (please complete the following information):**
 - MacOS 10.15.6
 - Version: flair-0.8



Discussion:

Resolving patch:
diff --git a/flair/trainers/trainer.py b/flair/trainers/trainer.py
--- a/flair/trainers/trainer.py
+++ b/flair/trainers/trainer.py
@@ -630,6 +630,8 @@ def train(
             final_score = 0
             log.info("Test data not provided setting final score to 0")
 
+        log_handler.close()
+
         log.removeHandler(log_handler)
 
         if self.use_tensorboard:


Test patch:
Repository: flairNLP/flair

Issue:
can't load ner-multi : 'LanguageModel' object has no attribute '_load_state_dict_pre_hooks'
**Describe the bug**
Multi lingual models (both ner or pos) fails to load after download inside torch nn.
I don't have any problems with other ner packages

**To Reproduce**

> from flair.data import Sentence
> from flair.models import SequenceTagger
> tagger = SequenceTagger.load("flair/ner-multi-fast")

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/flair/flair/nn.py", line 93, in load
    model = cls._init_model_with_state_dict(state)
  File "/home/flair/flair/models/sequence_tagger_model.py", line 297, in _init_model_with_state_dict
    model.load_state_dict(state["state_dict"])
  File "/root/anaconda3/envs/flair/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1037, in load_state_dict
    load(self)
  File "/root/anaconda3/envs/flair/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1035, in load
    load(child, prefix + name + '.')
  File "/root/anaconda3/envs/flair/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1035, in load
    load(child, prefix + name + '.')
  File "/root/anaconda3/envs/flair/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1035, in load
    load(child, prefix + name + '.')
  File "/root/anaconda3/envs/flair/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1031, in load
    module._load_from_state_dict(
  File "/root/anaconda3/envs/flair/lib/python3.9/site-packages/torch/nn/modules/module.py", line 957, in _load_from_state_dict
    for hook in self._load_state_dict_pre_hooks.values():
  File "/root/anaconda3/envs/flair/lib/python3.9/site-packages/torch/nn/modules/module.py", line 778, in __getattr__
    raise ModuleAttributeError("'{}' object has no attribute '{}'".format(
torch.nn.modules.module.ModuleAttributeError: 'LanguageModel' object has no attribute '_load_state_dict_pre_hooks'
```
 - tried on macos 11.1, python 3.8 and python 3.6 and 3.9.4
 - torch 1.7.1 (conda env from requirements)
 - Version flair 0.8.2 (I tried both pip install and pip from github repo)




Discussion:
Hm that's strange. I just checked and it works in Flair 0.7. Have to take a closer look.
Thanks ! it works fine with flair 0.8.0

Resolving patch:
diff --git a/flair/models/language_model.py b/flair/models/language_model.py
--- a/flair/models/language_model.py
+++ b/flair/models/language_model.py
@@ -445,7 +445,7 @@ def __setstate__(self, d):
             self.eval()
 
         else:
-            self.__dict__ = d
+            super().__setstate__(d)
 
     def _apply(self, fn):
 


Test patch: