Pipelines support
Author: AlexanderKUA
Pull request: no
Labels: name: Core: Pipeline

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):

I'm using roberta-base model for feature extraction through pipeline functionality.

Language I am using the model on English texts.

The problem arise when using:
* [x ] my own modified scripts: (give details)

```
from transformers import pipeline
import torch
#torch.set_default_tensor_type('torch.cuda.FloatTensor')

nlp = pipeline('feature-extraction', model='roberta-base', tokenizer='roberta-base', device=0)

def encode(input):
    with nlp.device_placement():
        return np.array(nlp(input)).mean(axis=1)

results = encode(['cybersecurity', 'cyber security', 'agriculture', 'data'])
```

## To Reproduce

Steps to reproduce the behavior:

1. Just run code above.

Error details

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-6-f0b27f7cd838> in <module>
----> 1 encode(['cybersecurity', 'cyber security', 'agriculture', 'data']).shape

<ipython-input-5-a0628a1cb908> in encode(input)
     12 def encode(input):
     13     with nlp.device_placement():
---> 14         return np.array(nlp(input)).mean(axis=1)

~/anaconda3/lib/python3.6/site-packages/transformers/pipelines.py in __call__(self, *args, **kwargs)
    442 
    443     def __call__(self, *args, **kwargs):
--> 444         return super().__call__(*args, **kwargs).tolist()
    445 
    446 

~/anaconda3/lib/python3.6/site-packages/transformers/pipelines.py in __call__(self, *texts, **kwargs)
    402             # Filter out features not available on specific models
    403             inputs = self.inputs_for_model(inputs)
--> 404             return self._forward(inputs)
    405 
    406     def _forward(self, inputs):

~/anaconda3/lib/python3.6/site-packages/transformers/pipelines.py in _forward(self, inputs)
    417         else:
    418             with torch.no_grad():
--> 419                 predictions = self.model(**inputs)[0].cpu()
    420 
    421         return predictions.numpy()

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask)
    733             head_mask = [None] * self.config.num_hidden_layers
    734 
--> 735         embedding_output = self.embeddings(input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds)
    736         encoder_outputs = self.encoder(embedding_output,
    737                                        attention_mask=extended_attention_mask,

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_roberta.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
     68                                                       token_type_ids=token_type_ids,
     69                                                       position_ids=position_ids,
---> 70                                                       inputs_embeds=inputs_embeds)
     71 
     72 

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
    184 
    185         if inputs_embeds is None:
--> 186             inputs_embeds = self.word_embeddings(input_ids)
    187         position_embeddings = self.position_embeddings(position_ids)
    188         token_type_embeddings = self.token_type_embeddings(token_type_ids)

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/sparse.py in forward(self, input)
    112         return F.embedding(
    113             input, self.weight, self.padding_idx, self.max_norm,
--> 114             self.norm_type, self.scale_grad_by_freq, self.sparse)
    115 
    116     def extra_repr(self):

~/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1482         # remove once script supports set_grad_enabled
   1483         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1484     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1485 
   1486 

RuntimeError: Expected object of device type cuda but got device type cpu for argument #3 'index' in call to _th_index_select
```
## Expected behavior
Sentences encoded properly.
## Environment

* OS: Ubuntu 18.01
* Python version: Python 3.6.5 :: Anaconda, Inc.
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): 2.3.0 and master
* Using GPU ? yes
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context
`torch.set_default_tensor_type('torch.cuda.FloatTensor')`
Uncommenting such line solves issue partially. Issue with CUDA tensor disappears but those sentences could not be encoded properly

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-10-f0b27f7cd838> in <module>
----> 1 encode(['cybersecurity', 'cyber security', 'agriculture', 'data']).shape

<ipython-input-9-138f4526e218> in encode(input)
     12 def encode(input):
     13     with nlp.device_placement():
---> 14         return np.array(nlp(input)).mean(axis=1)

~/anaconda3/lib/python3.6/site-packages/transformers/pipelines.py in __call__(self, *args, **kwargs)
    442 
    443     def __call__(self, *args, **kwargs):
--> 444         return super().__call__(*args, **kwargs).tolist()
    445 
    446 

~/anaconda3/lib/python3.6/site-packages/transformers/pipelines.py in __call__(self, *texts, **kwargs)
    402             # Filter out features not available on specific models
    403             inputs = self.inputs_for_model(inputs)
--> 404             return self._forward(inputs)
    405 
    406     def _forward(self, inputs):

~/anaconda3/lib/python3.6/site-packages/transformers/pipelines.py in _forward(self, inputs)
    417         else:
    418             with torch.no_grad():
--> 419                 predictions = self.model(**inputs)[0].cpu()
    420 
    421         return predictions.numpy()

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask)
    733             head_mask = [None] * self.config.num_hidden_layers
    734 
--> 735         embedding_output = self.embeddings(input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds)
    736         encoder_outputs = self.encoder(embedding_output,
    737                                        attention_mask=extended_attention_mask,

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_roberta.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
     68                                                       token_type_ids=token_type_ids,
     69                                                       position_ids=position_ids,
---> 70                                                       inputs_embeds=inputs_embeds)
     71 
     72 

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
    188         token_type_embeddings = self.token_type_embeddings(token_type_ids)
    189 
--> 190         embeddings = inputs_embeds + position_embeddings + token_type_embeddings
    191         embeddings = self.LayerNorm(embeddings)
    192         embeddings = self.dropout(embeddings)

RuntimeError: CUDA error: device-side assert triggered
```
even with
CUDA_LAUNCH_BLOCKING=1
If we try to encode sentence by sentence everything works.


Comments:
Hi @AlexanderKUA, thanks for reporting this issue.

Can you give a try to the following commit 088daf78d45bed144fe2af84b538f573573bd01d and let us know if it solves your issue ?

```python
from transformers import pipeline
nlp = pipeline('feature-extraction', model='distilbert-base-uncased', device=0)
print(nlp(['cybersecurity', 'cyber security', 'agriculture', 'data']))
```

Thanks, 
Morgan

Hi @mfuntowicz 
I checked your commit. Yes, it solves the issue. Thanks a lot.