Pipeline(summarization): CUDA error: an illegal memory access was encountered
Author: avacaondata
Pull request: no
Labels: name: Core: Pipeline

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.3.1
- Platform: Ubuntu 18
- Python version: 3.7
- PyTorch version (GPU?): 1.5.0 (YES)
- Tensorflow version (GPU?): 2.2.0 (YES)
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: NO

### Who can help

@sshleifer 

## Information

Model I am using (Bert, XLNet ...): BART for Summarization (pipeline)

The problem arises when using:
```python
class Summarizer:
    def __init__(self, min_length: int = 10, max_length: int = 120, device=1):
        self.summarizer = pipeline("summarization", device=device)
        self.min_length=min_length
        self.max_length=max_length
    
    def summarize(self, articles_df):
        empty_abstract = articles_df[articles_df["abstract"] == ""]
        for idx, paper in tqdm(empty_abstract.iterrows(), desc="Iterating over papers with empty abstract"):
            articles_df.loc[idx, "abstract"] = self._summarize_one_paper(paper)
        return articles_df
    
    def _summarize_one_paper(self, paper):
        try:
            summaries = self.summarizer(paper["paragraphs"], min_length=self.min_length, max_length=self.max_length)
        except:
            summaries = []
            for paragraph in paper["paragraphs"]:
                summaries.extend(
                    self.summarizer(paragraph, min_length=self.min_length, max_length=self.max_length)
                )
        return ["".join([summary["summary_text"] for summary in summaries])]
summarizer = Summarizer()
data_summarized = summarizer.summarize(data)
```

The tasks I am working on is:
* Summarization

The error arises after 848 iterations, which is what surprises me the most... It can access CUDA up until that point.

```
RuntimeError                              Traceback (most recent call last)
<ipython-input-5-098ee40477ee> in <module>
----> 1 data_summarized = summarizer.summarize(data)

<ipython-input-3-671b9c11b7bf> in summarize(self, articles_df)
      8         empty_abstract = articles_df[articles_df["abstract"] == ""]
      9         for idx, paper in tqdm(empty_abstract.iterrows(), desc="Iterating over papers with empty abstract"):
---> 10             articles_df.loc[idx, "abstract"] = self._summarize_one_paper(paper)
     11         return articles_df
     12 

<ipython-input-3-671b9c11b7bf> in _summarize_one_paper(self, paper)
     18             for paragraph in paper["paragraphs"]:
     19                 summaries.extend(
---> 20                     self.summarizer(paragraph, min_length=self.min_length, max_length=self.max_length)
     21                 )
     22         return ["".join([summary["summary_text"] for summary in summaries])]

~/miniconda/envs/transformers_env/lib/python3.7/site-packages/transformers/pipelines.py in __call__(self, return_tensors, return_text, clean_up_tokenization_spaces, *documents, **generate_kwargs)
   1926 
   1927             if self.framework == "pt":
-> 1928                 inputs = self.ensure_tensor_on_device(**inputs)
   1929                 input_length = inputs["input_ids"].shape[-1]
   1930             elif self.framework == "tf":

~/miniconda/envs/transformers_env/lib/python3.7/site-packages/transformers/pipelines.py in ensure_tensor_on_device(self, **inputs)
    601             :obj:`Dict[str, torch.Tensor]`: The same as :obj:`inputs` but on the proper device.
    602         """
--> 603         return {name: tensor.to(self.device) for name, tensor in inputs.items()}
    604 
    605     def check_model_type(self, supported_models: Union[List[str], dict]):

~/miniconda/envs/transformers_env/lib/python3.7/site-packages/transformers/pipelines.py in <dictcomp>(.0)
    601             :obj:`Dict[str, torch.Tensor]`: The same as :obj:`inputs` but on the proper device.
    602         """
--> 603         return {name: tensor.to(self.device) for name, tensor in inputs.items()}
    604 
    605     def check_model_type(self, supported_models: Union[List[str], dict]):

RuntimeError: CUDA error: an illegal memory access was encountered
```



Comments:
I would guess that that traceback is either masking OOM or IndexError. You will get a better traceback if you try on CPU.