Repository: deepset-ai/haystack

Issue:
RAG Predict throwing an error when using a GPU
**Describe the bug**
I have no issue creating the generator, but creating it with use_gpu on will lead to an error when attempting to predict. Note that I have no issues with the FARMReader when using a gpu.

**Error message**

```
Inferencing Samples: 100%|██████████| 1/1 [00:00<00:00, 56.41 Batches/s]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-17-42737cee0272> in <module>()
     10         question=question,
     11         documents=retriever_results,
---> 12         top_k=1
     13     )
     14 

/usr/local/lib/python3.6/dist-packages/haystack/generator/transformers.py in predict(self, question, documents, top_k)
    236         # Compute doc scores from docs_embedding
    237         doc_scores = torch.bmm(question_embedding.unsqueeze(1),
--> 238                                passage_embeddings.unsqueeze(0).transpose(1, 2)).squeeze(1)
    239 
    240         # TODO Need transformers 3.4.0

RuntimeError: Expected object of device type cuda but got device type cpu for argument #0 'result' in call to _th_bmm_out
```

**Additional context**
This is in google colab, I can't test this anywhere else as I don't have access to a GPU.
I did make sure to setup torch at the top:
```
!pip install urllib3==1.25.4 
!pip install torch==1.6.0+cu101 torchvision==0.6.1+cu101 -f https://download.pytorch.org/whl/torch_stable.html
```
Without this torch update, it doesn't throw the error, but it doesn't appear to be using the GPU as it greatly improves the speed of the DPR (Which also has use_gpu on). 

**To Reproduce**
Should be able to reproduce by taking the RAG tutorial and flipping off to on for use_gpu

**System:**
 - OS: Google Colab
 - GPU/CPU: Colab GPU
 - Haystack version (commit or version number): Latest
 - DocumentStore: FAISSDocumentStore
 - Retriever: DensePassageRetriever



Discussion:
Thanks for reporting @rshtirmer !
Seems like one of the tensors is still on the CPU. We'lll look into this.
I have prepared a fix for it. Just testing on colab then share PR

Resolving patch:
diff --git a/haystack/generator/transformers.py b/haystack/generator/transformers.py
--- a/haystack/generator/transformers.py
+++ b/haystack/generator/transformers.py
@@ -90,7 +90,6 @@ def __init__(
         """
 
         self.model_name_or_path = model_name_or_path
-        self.top_k_answers = top_k_answers
         self.max_length = max_length
         self.min_length = min_length
         self.generator_type = generator_type
@@ -99,8 +98,15 @@ def __init__(
         self.prefix = prefix
         self.retriever = retriever
 
+        if top_k_answers > self.num_beams:
+            top_k_answers = self.num_beams
+            logger.warning(f'top_k_answers value should not be greater than num_beams, hence setting it to {num_beams}')
+
+        self.top_k_answers = top_k_answers
+
         if use_gpu and torch.cuda.is_available():
             self.device = torch.device("cuda")
+            raise AttributeError("Currently RAGenerator does not support GPU, try with use_gpu=False")
         else:
             self.device = torch.device("cpu")
 
@@ -202,6 +208,11 @@ def predict(self, question: str, documents: List[Document], top_k: Optional[int]
 
         top_k_answers = top_k if top_k is not None else self.top_k_answers
 
+        if top_k_answers > self.num_beams:
+            top_k_answers = self.num_beams
+            logger.warning(f'top_k_answers value should not be greater than num_beams, '
+                           f'hence setting it to {top_k_answers}')
+
         # Flatten the documents so easy to reference
         flat_docs_dict: Dict[str, Any] = {}
         for document in documents:


Test patch: