Repository: deepset-ai/haystack

Issue:
Tutorial 2 - RuntimeError: CUDA error: device-side assert triggered
When I try running tutorial 2 on Colab I run into this error message:
```console
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
```


This problem occurs from this line of code:
```python
reader = FARMReader(model_name_or_path="distilbert-base-uncased-distilled-squad", use_gpu=True)
```


Here's the output before the error occurs:
```console
INFO - haystack.modeling.utils -  Using devices: CUDA
INFO - haystack.modeling.utils -  Number of GPUs: 1
INFO - haystack.modeling.model.language_model -  LOADING MODEL
INFO - haystack.modeling.model.language_model -  =============
INFO - haystack.modeling.model.language_model -  Could not find distilbert-base-uncased-distilled-squad locally.
INFO - haystack.modeling.model.language_model -  Looking on Transformers Model Hub (in local cache and online)...
INFO - haystack.modeling.model.language_model -  Loaded distilbert-base-uncased-distilled-squad
```

Interestingly, the reader is loaded properly when using a CPU in the notebook settings.


**EDIT**
I solved the above issue by including this code in the notebook:
```python
# In Colab / No Docker environments: Start Elasticsearch from source
! wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.2-linux-x86_64.tar.gz -q
! tar -xzf elasticsearch-7.9.2-linux-x86_64.tar.gz
! chown -R daemon:daemon elasticsearch-7.9.2

import os
from subprocess import Popen, PIPE, STDOUT
es_server = Popen(['elasticsearch-7.9.2/bin/elasticsearch'],
                   stdout=PIPE, stderr=STDOUT,
                   preexec_fn=lambda: os.setuid(1)  # as daemon
                  )
# wait until ES has started
! sleep 30
```

But, I still run into the same error message, now from this line:
```python
reader.train(data_dir=data_dir, train_filename="dev-v2.0.json", use_gpu=True, n_epochs=1, save_dir="my_model")
```


Discussion:
Hi @ArminBaz my first guess is that the notebook is running out of GPU memory. This can sometimes happen when you rerun some of the cells but the memory is not properly released beforehand.
I will try to reproduce the error and investigate what's causing the issue.
For debugging purposes I added the following to lines at the top of the notebook of tutorial 2.
```
import os
os.environ['CUDA_LAUNCH_BLOCKING'] = "1"
```

I also get the CUDA error: device-side assert triggered with the very first training batch:
```
Train epoch 0/0 (Cur. train loss: 0.0000):   0%|          | 0/1360 [00:00<?, ?it/s]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-5-6966fdb1398c> in <module>()
      2 data_dir = "data/squad20"
      3 # data_dir = "PATH/TO_YOUR/TRAIN_DATA"
----> 4 reader.train(data_dir=data_dir, train_filename="dev-v2.0.json", use_gpu=True, n_epochs=1, save_dir="my_model")

9 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing)
   2844     if size_average is not None or reduce is not None:
   2845         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2846     return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
   2847 
   2848 

RuntimeError: CUDA error: device-side assert triggered
```

Maybe it has to do with our recent torch upgrade. I will try an earlier release of haystack via replacing the line
`!pip install git+https://github.com/deepset-ai/haystack.git`with
`!pip install farm-haystack`
Hi @ArminBaz I think the problem is caused by our upgrade to pytorch 1.10. I can get rid of the problem by downgrading the torch dependency to 1.9 (and also downgrading torch-vision). For example, you can do that by changing the following line in the requirements.txt before installing haystack's dependencies in a virtual environment:
`torch>1.5,<1.11` to
`torch>1.5,<1.10`
That solved the problem for me.


I will continue to investigate the error with torch 1.10. Parts of the reader node are the problem.
 ```
 File "/haystack/haystack/modeling/model/prediction_head.py", line 426, in logits_to_preds
    start_end_matrix[:, indices[0][:], indices[1][:]] = -888
 ```

Resolving patch:
diff --git a/haystack/modeling/model/prediction_head.py b/haystack/modeling/model/prediction_head.py
--- a/haystack/modeling/model/prediction_head.py
+++ b/haystack/modeling/model/prediction_head.py
@@ -342,6 +342,13 @@ def logits_to_loss(self, logits: torch.Tensor, labels: torch.Tensor, **kwargs):
         start_position.clamp_(0, ignored_index)
         end_position.clamp_(0, ignored_index)
 
+        # Workaround for pytorch bug in version 1.10.0 with non-continguous tensors
+        # Fix expected in 1.10.1 based on https://github.com/pytorch/pytorch/pull/64954
+        start_logits=start_logits.contiguous()
+        start_position=start_position.contiguous()
+        end_logits=end_logits.contiguous()
+        end_position=end_position.contiguous()
+        
         loss_fct = CrossEntropyLoss(reduction="none")
         start_loss = loss_fct(start_logits, start_position)
         end_loss = loss_fct(end_logits, end_position)
@@ -1047,4 +1054,4 @@ def _is_json(x):
         json.dumps(x)
         return True
     except:
-        return False
\ No newline at end of file
+        return False


Test patch: