Repository: flairNLP/flair

Issue:
Problem with loading dialogpt
**Describe the bug**
```
----------------------------------------------------------------------------------------------------
2020-06-12 18:44:49,152 Model training base path: "dialogpt"
2020-06-12 18:44:49,152 ----------------------------------------------------------------------------------------------------

2020-06-12 18:44:49,153 Device: cuda:0
2020-06-12 18:44:49,154 ----------------------------------------------------------------------------------------------------
2020-06-12 18:44:49,155 Embeddings storage mode: cpu
2020-06-12 18:44:49,230 ----------------------------------------------------------------------------------------------------
2020-06-12 18:49:00,669 epoch 1 - iter 76/766 - loss 1.11887558 - samples/sec: 9.67

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-7-e12f92b71d0b> in <module>
      7         mini_batch_chunk_size=2, # set this if you get OOM errors
      8         max_epochs=1, # very few epochs of fine-tuning
----> 9         embeddings_storage_mode='cpu',
     10         #checkpoint=True
     11     )

~/venv36/lib64/python3.6/site-packages/flair/trainers/trainer.py in train(self, base_path, learning_rate, mini_batch_size, mini_batch_chunk_size, max_epochs, scheduler, anneal_factor, patience, initial_extra_patience, min_learning_rate, train_with_dev, monitor_train, monitor_test, embeddings_storage_mode, checkpoint, save_final_model, anneal_with_restarts, anneal_with_prestarts, batch_growth_annealing, shuffle, param_selection_mode, num_workers, sampler, use_amp, amp_opt_level, eval_on_train_fraction, eval_on_train_shuffle, **kwargs)
    376                         if not param_selection_mode:
    377                             weight_extractor.extract_weights(
--> 378                                 self.model.state_dict(), iteration
    379                             )
    380 

~/venv36/lib64/python3.6/site-packages/flair/training_utils.py in extract_weights(self, state_dict, iteration)
    262             vec = state_dict[key]
    263             weights_to_watch = min(
--> 264                 self.number_of_weights, reduce(lambda x, y: x * y, list(vec.size()))
    265             )
    266 

TypeError: reduce() of empty sequence with no initial value

```

**To Reproduce**
Steps to reproduce the behavior (e.g. which model did you train? what parameters did you use? etc.).

Take a tagging corpus of choice and run:

```
embeddings = TransformerWordEmbeddings(
        'microsoft/DialoGPT-small',
        layers="-1",
        pooling_operation='first_last',
        fine_tune=True,
    )

tagger: SequenceTagger = SequenceTagger(
        hidden_size=256,
        embeddings=embeddings,
        tag_dictionary=tag_dictionary,
        tag_type=TAG_NAME,
        use_crf=False,
        use_rnn=False,
    )

trainer = ModelTrainer(tagger, corpus, optimizer=torch.optim.Adam)

trainer.train('dialogpt',
        learning_rate=3e-5,
        mini_batch_chunk_size=2,
        max_epochs=1,
        embeddings_storage_mode='cpu'
    )


```

**Expected behavior**
The models should train, instead I get the error.

**Environment (please complete the following information):**
 - Centos7
 - Flair 0.5.0



Discussion:
The same error happens with `distilgpt2`. The `ctrl` model also does not work (embedding tensor is 0-sized).
IMO these models are not suitable for this task and are not supported by transformers for tagging and classification.
Also encountering this with GPT2 for TextClassificationModel
`vec = state_dict[key]
             weights_to_watch = min(
                                  self.number_of_weights, reduce(lambda x, y: x * y, list(vec.size()))
                           )`
I think the error is occurring as the list in this code is remaining empty somehow for these transformers. So, if we are able to add an exception here for these transformers, it will work ig.  So, from what I can see in the above comments, does that mean Flair seems to have this issue for GPT transformers?

BTW, one interesting thing I found was on setting `param_selection_mode = True` in trainer.train(), we are able to avoid the bug (I have checked it for Distilgpt2. But, nevertheless this should be fixed. 
I'm not sure that setting ` param_selection_mode = True` you truly "avoided" the bug. My guess is that most probably you're avoiding a specific sentence that is causing trouble (which maybe was assigned to the test set). If it is a problem of missing embeddings for that sentence, I have had the same issue with BERT in #1703 .
Hmm, it is not a problem of missing embeddings. What I observed is, whenever I use a GPT model like GPT2, DistilGPT2, DialoGPT, I encounter the above error. This error is solved everytime I use the setting `param_selection_mode = True`, now this setting tests the model against dev set. I guess you might be right when you said that I have not truly avoided the bug. It will be pretty interesting to figure out what is happening.

Resolving patch:
diff --git a/flair/embeddings/token.py b/flair/embeddings/token.py
--- a/flair/embeddings/token.py
+++ b/flair/embeddings/token.py
@@ -821,7 +821,7 @@ def __init__(
         batch_size: int = 1,
         use_scalar_mix: bool = False,
         fine_tune: bool = False,
-        allow_long_sentences: bool = False,
+        allow_long_sentences: bool = True,
         **kwargs
     ):
         """
@@ -854,7 +854,7 @@ def __init__(
             self.max_subtokens_sequence_length = self.tokenizer.model_max_length
             self.stride = self.tokenizer.model_max_length//2
         else:
-            self.max_subtokens_sequence_length = None
+            self.max_subtokens_sequence_length = self.tokenizer.model_max_length
             self.stride = 0
 
         # model name
diff --git a/flair/training_utils.py b/flair/training_utils.py
--- a/flair/training_utils.py
+++ b/flair/training_utils.py
@@ -262,9 +262,13 @@ def extract_weights(self, state_dict, iteration):
         for key in state_dict.keys():
 
             vec = state_dict[key]
-            weights_to_watch = min(
-                self.number_of_weights, reduce(lambda x, y: x * y, list(vec.size()))
-            )
+            # print(vec)
+            try:
+                weights_to_watch = min(
+                    self.number_of_weights, reduce(lambda x, y: x * y, list(vec.size()))
+                )
+            except:
+                continue
 
             if key not in self.weights_dict:
                 self._init_weights_index(key, state_dict, weights_to_watch)


Test patch: