Repository: flairNLP/flair

Issue:
TransformerDocuementEmbedding does not take flair.device
By setting `flair.device = torch.device('cuda:0;)`, it throws an error saying `expected device cpu  but got device cuda:0`. I have to explicitly set `TextClassifier(...).to(torch.device('cuda:0')` to make it happen


Discussion:
Hm that should not happen since the `TextClassifier` moves itself to flair.device when calling the constructor. Can you post a full code example to reproduce?
There:

```
import torch
from torch.optim.adam import Adam

from flair.datasets import CSVClassificationCorpus, ClassificationCorpus
from flair.data import Corpus
from flair.embeddings import (WordEmbeddings, FlairEmbeddings, TransformerDocumentEmbeddings )
from flair.models import TextClassifier
from flair.trainers import ModelTrainer

flair.device = torch.device('cuda:0')
flair.cache_root = './cache'

document_embeddings = TransformerDocumentEmbeddings('bert-base-uncased', use_scalar_mix=True)

corpus: Corpus = ClassificationCorpus(data_folder,
                                          test_file='test.txt',
                                          dev_file='val.txt',
                                          train_file='train.txt')

label_dict = corpus.make_label_dictionary()

classifier = TextClassifier(document_embeddings, 
                                label_dictionary=label_dict, 
                                multi_label=False))

trainer = ModelTrainer(classifier, corpus, optimizer=Adam, use_tensorboard=True)

trainer.train(result_folder,
                  max_epochs=200, 
                  embeddings_storage_mode='gpu',
                  learning_rate = 1e-3,
                  mini_batch_size = 32,
    )
```
Thanks! Can you also verify that your GPU is visible? 

Does the following print out True?
```python
import torch
print(torch.cuda.is_available())
```
Yes - I've been training using Flair's document embedding and text classification for multiple tasks..

<img width="317" alt="image" src="https://user-images.githubusercontent.com/3516360/80873789-e3c91c80-8cec-11ea-91bc-c1d79395a9f6.png">

Hello @junhua thanks for sharing the script - the error comes from setting `use_scalar_mix=True`. This option should only be used if not fine-tuning document transformer embeddings. But since you are fine-tuning, you should use `use_scalar_mix=False`. 

Still, we have to change this so that this error does not occur. I'll investigate and do a PR.

Resolving patch:
diff --git a/flair/embeddings.py b/flair/embeddings.py
--- a/flair/embeddings.py
+++ b/flair/embeddings.py
@@ -997,14 +997,23 @@ def __init__(self, mixture_size: int, trainable: bool = False) -> None:
         self.scalar_parameters = ParameterList(
             [
                 Parameter(
-                    torch.FloatTensor([initial_scalar_parameters[i]]),
+                    torch.tensor(
+                        [initial_scalar_parameters[i]],
+                        dtype=torch.float,
+                        device=flair.device,
+                    ),
                     requires_grad=trainable,
+
                 )
                 for i in range(mixture_size)
             ]
         )
         self.gamma = Parameter(
-            torch.FloatTensor([1.0]), requires_grad=trainable
+            torch.tensor(
+                [1.0],
+                dtype=torch.float,
+                device=flair.device,
+            ), requires_grad=trainable
         )
 
     def forward(self, tensors: List[torch.Tensor]) -> torch.Tensor:


Test patch: