Repository: flairNLP/flair

Issue:
Add better sentiment analysis model
With the upcoming #1492 we will add fine-tuneable transformers to Flair, yielding much improved classification performance. We will use the opportunity to replace the current sentiment analysis model in Flair with a better one, trained over more data and with a BERT-style architecture. To add this model, we should: 

- [x]  Add new sentiment analysis datasets to Flair. Currently we have the IMDB and SentEval datasets, but we should add datasets for other domains beside movie reviews. 
- [x] Train strong model over aggregated sentiment datasets
- [x] Add model to Flair for download 


Discussion:
I trained a few models for sentiment analysis in different ways:

1. FastText word embeddings + LSTM - training with SGD with annealing for max 150 epochs
2. Transformers (BERT and RoBERTa) - fine-tuning with Adam for 5 epochs

In both cases, the best model is selected using holdout DEV data and evaluated on holdout TEST data. 

Here are the results: 

| Model                      | Test F1 |
|----------------------------|---------|
| FastText + LSTM            | **92.59**   |
| BERT (base uncased)        | 93.14   |
| BERT (base cased)          | 93.06   |
| DistillBERT (base uncased) | 92.82   |
| DistillBERT (base cased)   | 92.69   |
| RoBERTa (base)             | **93.43**   |
| DistilRoBERTa (base)       | 93.03   |
The models were trained over a combination of 5 different sentiment analysis corpora: 

```python
corpus = MultiCorpus([
    IMDB(filter_if_longer_than=50),
    SENTEVAL_SST_BINARY(filter_if_longer_than=50),
    SENTEVAL_MR(filter_if_longer_than=50),
    SENTIMENT_140().downsample(0.1, downsample_test=False),
    AMAZON_REVIEWS(filter_if_longer_than=50, memory_mode='partial')
]
)
```

resulting in a very large training corpus of some 400.000 text data points. There were three classes: POSITIVE, NEGATIVE and NEUTRAL. However, the NEUTRAL class gets lowest scores, see output of RoBERTa model: 

```console
MICRO_AVG: acc 0.9562032772030507 - f1-score 0.934304915804576
MACRO_AVG: acc 0.9562032772030507 - f1-score 0.7438650886414663
NEGATIVE   tp: 4457 - fp: 665 - fn: 392 - tn: 20972 - precision: 0.8702 - recall: 0.9192 - accuracy: 0.9601 - f1-score: 0.8940
NEUTRAL    tp: 302 - fp: 191 - fn: 831 - tn: 25162 - precision: 0.6126 - recall: 0.2665 - accuracy: 0.9614 - f1-score: 0.3715
POSITIVE   tp: 19987 - fp: 884 - fn: 517 - tn: 5098 - precision: 0.9576 - recall: 0.9748 - accuracy: 0.9471 - f1-score: 0.9661
```

This is likely because the NEUTRAL class exists only in one of the datasets (Amazon reviews with 3 stars). I'll rerun with 3 star reviews mapped to NEGATIVE.

Resolving patch:
diff --git a/flair/models/text_classification_model.py b/flair/models/text_classification_model.py
--- a/flair/models/text_classification_model.py
+++ b/flair/models/text_classification_model.py
@@ -431,9 +431,8 @@ def _labels_to_indices(self, sentences: List[Sentence]):
     def _fetch_model(model_name) -> str:
 
         model_map = {}
-        aws_resource_path = (
-            "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/models-v0.4"
-        )
+        aws_resource_path = "https://s3.eu-central-1.amazonaws.com/alan-nlp/resources/models-v0.4"
+        hu_path: str = "https://nlp.informatik.hu-berlin.de/resources/models"
 
         model_map["de-offensive-language"] = "/".join(
             [
@@ -443,8 +442,15 @@ def _fetch_model(model_name) -> str:
             ]
         )
 
+        # English sentiment models
+        model_map["sentiment"] = "/".join(
+            [hu_path, "sentiment-curated-distilbert", "sentiment-en-mix-distillbert.pt"]
+        )
         model_map["en-sentiment"] = "/".join(
-            [aws_resource_path, "classy-imdb-en-rnn-cuda%3A0", "imdb-v0.4.pt"]
+            [hu_path, "sentiment-curated-distilbert", "sentiment-en-mix-distillbert.pt"]
+        )
+        model_map["sentiment-fast"] = "/".join(
+            [hu_path, "sentiment-curated-fasttext-rnn", "sentiment-en-mix-ft-rnn.pt"]
         )
 
         cache_dir = Path("models")


Test patch: