Repository: flairNLP/flair

Issue:
How to specify CPU or GPU?
I'd like to know how to move the model between CPU and GPU. Alternatively, I'd like to know if it's possible to specify the device before instantiating a model. 


Discussion:
Hello @romanovzky the default behavior is that the model gets put on GPU if available and runs on CPU if there is no GPU. 

This is done in the `__init__.py`, here: 

```python
device = None
if torch.cuda.is_available():
    device = torch.device('cuda:0')
else:
    device = torch.device('cpu')
```

The `flair.device` parameter gets called all over the code to move models and tensor to the device on which flair is run.

So if you would like to explicitly change this behavior, for instance to direct it to run on CPU even if you have a GPU available, you need to run this code before instantiating your model: 

```python
import flair, torch
flair.device = torch.device('cpu') 
```

This overwrites the `flair.device` with your desired destination. 

Hope this clarifies!  




Closing since question is answered (hopefully) - feel free to reopen if you have other questions.
Hi Alan,

I'm running the language model trainer on a server with 2 GPUs. Both GPUs are available. It works on GPU 0, however, when I try to change the device to GPU 1, I face the following error:

> Traceback (most recent call last):
  File "train_LM_flair.py", line 42, in <module>
    trainer.train('/home/tobdau/data/language_model1', learning_rate= 20, mini_batch_size=100, sequence_length=250, patience=20, max_epochs=20, anneal_factor=0.999)
  File "/home/tobdau/.local/lib/python3.6/site-packages/flair/trainers/language_model_trainer.py", line 336, in train
    log.info(self.model.generate_text())
  File "/home/tobdau/.local/lib/python3.6/site-packages/flair/models/language_model.py", line 259, in generate_text
    prediction, _, hidden = self.forward(input, hidden)
  File "/home/tobdau/.local/lib/python3.6/site-packages/flair/models/language_model.py", line 71, in forward
    encoded = self.encoder(input)
  File "/home/tobdau/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tobdau/.local/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 118, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/home/tobdau/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 1454, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: arguments are located on different GPUs at /pytorch/aten/src/THC/generic/THCTensorIndex.cu:519

I use the following code:

`flair.device = torch.device('cuda:1')`

Best,
Tobias
Hello @TDaudert - thanks for spotting and reporting this. The error is likely here: 

https://github.com/zalandoresearch/flair/blob/11850eeac9dcfaa89410f6748a9c71e56d331492/flair/models/language_model.py#L277-L278 

i.e., we only check if cuda is available and if so then put on cuda:0. But in the rest of the code, we use the newer .to(device) method, like here: 

https://github.com/zalandoresearch/flair/blob/11850eeac9dcfaa89410f6748a9c71e56d331492/flair/models/language_model.py#L128

I will put in a PR which should fix this error!



Resolving patch:
diff --git a/flair/models/language_model.py b/flair/models/language_model.py
--- a/flair/models/language_model.py
+++ b/flair/models/language_model.py
@@ -273,9 +273,7 @@ def generate_text(
                         .unsqueeze(0)
                     )
 
-                input = torch.cat(char_tensors)
-                if torch.cuda.is_available():
-                    input = input.cuda()
+                input = torch.cat(char_tensors).to(flair.device)
 
                 prediction, _, hidden = self.forward(input, hidden)
 
@@ -289,8 +287,7 @@ def generate_text(
 
             for i in range(number_of_characters):
 
-                if torch.cuda.is_available():
-                    input = input.cuda()
+                input = input.to(flair.device)
 
                 # get predicted weights
                 prediction, _, hidden = self.forward(input, hidden)


Test patch: