Repository: flairNLP/flair

Issue:
Model for Fine-grained POS Tagging of German Tweets
Hi,

recently I received the training, dev and test data from the "Fine-grained POS Tagging of German Tweets" paper (which can be found [here](https://pdfs.semanticscholar.org/82c9/90aa15e2e35de8294b4a721785da1ede20d0.pdf)). The highest accuracy mentioned in that paper is 89.42%.

I trained a model with German fasttext embeddings and forward/backward lm embeddings and after 40 epochs (on CPU) an accuracy ~ 92% could be achieved. I double checked that result with an own evaluation script that loads the trained model, predicts the tags and compares the predicted tags with the gold standard and this also achieves 92%.

So I would like to ask if the core team is interested in such a pos tagging model for German twitter data? I would really like to share the model, but then I have to ask the paper author for a permission to share a trained model.


Discussion:
Hi Stefan, 

wow that's great! Did the training terminate after the annealing rate leveled off or did you set a fixed set of epochs? Could you also post the training parameters? 

We're absolutely interested in including this if possible. The easiest would be to report the results on the comparative results table in the README, but we could also think about packaging the model to that it becomes loadable as a standard pre-trained model. Do you think a model trained over this relatively small dataset would be helpful to people, or do you think the standard POS model for German is enough?
Hi Alan,

I wanted to apply the `de-pos` model on the test data, but then I recognized, that the tag set differs (STTS vs Universal Tagset). The UD dataset includes the STTS tags in the column next to the universal pos tag, so I'm currently retraining a model. I think we then have a good comparison. 

Btw: the Twitter dataset comes with additional pos tags (support for hashtags, emoticons...) 
Hi Alan,

the training parameters were:

```python
tagger: SequenceTagger = SequenceTagger(hidden_size=256,
                                        embeddings=embeddings,
                                        tag_dictionary=tag_dictionary,
                                        tag_type=tag_type,
                                        use_crf=True)
```

For the `SequenceTagger`. I used `WordEmbeddings('de-fasttext')`, `CharLMEmbeddings('german-forward')` and `CharLMEmbeddings('german-backward')` as embeddings. Training was started with:

```bash
trainer.train('resources/taggers/example-ner',
              learning_rate=0.1,
              mini_batch_size=32,
              max_epochs=150,
              save_model=True)
```
I trained for 150 epochs and achieve an accuracy of **92,43**% - the original paper achieved 89,42%.


I trained on the UD dataset with STTS tags and evaluated on the Twitter test data. It achieved an accuracy of ~ 80%.

And now the good news: I got the permission to release a trained model from the author of that paper! So if you want I can send you / upload the model and we can integrate it into the upcoming release of `flair` :)

@stefan-it We would be interested in adding your model to the next flair release. Could you share the model with us? Many thanks in advance!
@tabergma I did some hyper parameter search with the newly introduces `hyperopt` interface and the accuracy is now 93.06 :)

The model can be found here:

```bash
wget https://schweter.eu/cloud/flair-models/de-pos-twitter-v0.1.pt
```

I will update my README [here](https://github.com/stefan-it/flair-experiments/tree/master/pos-twitter-german) soon. The model was trained with PyTorch in version 1.0 with CUDA 10.

Feel free to include the model in the upcoming release :)

Resolving patch:
diff --git a/flair/models/sequence_tagger_model.py b/flair/models/sequence_tagger_model.py
--- a/flair/models/sequence_tagger_model.py
+++ b/flair/models/sequence_tagger_model.py
@@ -629,6 +629,12 @@ def load(model: str):
                                   'de-pos-ud-v0.2.pt'])
             model_file = cached_path(base_path, cache_dir=cache_dir)
 
+        elif model.lower() == 'de-pos-fine-grained':
+            base_path = '/'.join([aws_resource_path_v04,
+                                  'POS-fine-grained-german-tweets',
+                                  'de-pos-twitter-v0.1.pt'])
+            model_file = cached_path(base_path, cache_dir=cache_dir)
+
         elif model.lower() == 'de-ner':
             base_path = '/'.join([aws_resource_path,
                                   'NER-conll03ger--h256-l1-b32-%2Bde-fasttext%2Bgerman-forward%2Bgerman-backward--v0.2',


Test patch:
diff --git a/tests/test_model_integration.py b/tests/test_model_integration.py
--- a/tests/test_model_integration.py
+++ b/tests/test_model_integration.py
@@ -406,7 +406,7 @@ def test_train_language_model(results_base_path, resources_path):
 
     text = language_model.generate_text(100)
     assert (text is not None)
-    assert (len(text) == 100)
+    assert (len(text) >= 100)
 
     # clean up results directory
     shutil.rmtree(results_base_path, ignore_errors=True)