Getting a signal: aborted (core dumped) or Segmentation fault (core dumped) error when trying to train on a single GPU
Author: dberma15
Pull request: no

## Environment info
- `transformers` version: 4.17.0
- Platform: Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.7
- PyTorch version (GPU?): 1.11.0+cu102 (False)
- Tensorflow version (GPU?): 2.8.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
- Wav2Vec2, HuBERT, SpeechEncoderDecoder, UniSpeech, UniSpeechSAT, SEW, SEW-D, Speech2Text: @patrickvonplaten, @anton-l


## Information

Model I am using Wav2Vec and I'm not able to train the model on GPU. Everytime I try, I get a core dump error and I cannot figure out why. The problem does not occur when I run the code without a GPU. I'm on a virtual machine using bitfusion so when I run the command bitfusion -n 1 python script.py but if I fun python script.py, it runs and trains the model:

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. run the code below using the gpu (bitfusion -n 1 python script.py) and I get either segmentation fault (core dumped) or aborted (core dumped)
2. run the code without python script.py and the code runs. 

```
import os
import pickle
import torch
import json
import time
import random
import re
import numpy as np
import pandas as pd
from datasets import Dataset, load_metric, load_dataset
from datasets import ClassLabel
from dataclasses import dataclass, field
from tqdm import tqdm
# from torch.utils.data import Dataset
from transformers import Wav2Vec2CTCTokenizer
from transformers import Wav2Vec2FeatureExtractor
from transformers import Wav2Vec2Processor
from transformers import Wav2Vec2ForCTC
from transformers import TrainingArguments
from transformers import Trainer
from transformers import Wav2Vec2Config

from typing import Dict, List, Optional, Union

def main():

    class wordSignalDataset(Dataset):
        def __init__(self, signal_file, wordfile,processor, padding):
            self.xdata = np.load(signal_file)
            self.ydata = np.load(wordfile)
            self.processor = processor
            self.padding = padding
            self.pad_to_multiple_of_label= None
            self.pad_to_multiple_of = None
        #
        def __len__(self):
            return len(self.xdata)
        #
        def __getitem__(self, idx):
            if torch.is_tensor(idx):
                idx = idx.tolist()
            batch = self.processor.pad([{'input_values':self.xdata[idx]}],
                            padding=self.padding, 
                            return_tensors='pt',
                            )
            print(batch)
            # batch['input_values'] = batch['input_values'].cuda()
            with self.processor.as_target_processor():
                labels_batch = self.processor.pad([{'input_ids':self.ydata[idx]}],
                                padding=self.padding,
                                return_tensors='pt',
                                )
            # #
            labels = labels_batch["input_ids"].masked_fill(labels_batch.attention_mask.ne(1), -100)
            batch["input_ids"] = labels
            # batch["input_ids"] = labels.cuda()
            # batch['attention_mask']=batch['attention_mask'].cuda()
            return batch

    @dataclass
    class DataCollatorCTCWithPadding:
        """
        Data collator that will dynamically pad the inputs received.
        Args:
            processor (:class:`~transformers.Wav2Vec2Processor`)
                The processor used for proccessing the data.
            padding (:obj:`bool`, :obj:`str` or :class:`~transformers.tokenization_utils_base.PaddingStrategy`, `optional`, defaults to :obj:`True`):
                Select a strategy to pad the returned sequences (according to the model's padding side and padding index)
                among:
                * :obj:`True` or :obj:`'longest'`: Pad to the longest sequence in the batch (or no padding if only a single
                  sequence if provided).
                * :obj:`'max_length'`: Pad to a maximum length specified with the argument :obj:`max_length` or to the
                  maximum acceptable input length for the model if that argument is not provided.
                * :obj:`False` or :obj:`'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of
                  different lengths).
            max_length (:obj:`int`, `optional`):
                Maximum length of the ``input_values`` of the returned list and optionally padding length (see above).
            max_length_labels (:obj:`int`, `optional`):
                Maximum length of the ``labels`` returned list and optionally padding length (see above).
            pad_to_multiple_of (:obj:`int`, `optional`):
                If set will pad the sequence to a multiple of the provided value.
                This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >=
                7.5 (Volta).
        """
        #
        processor: Wav2Vec2Processor
        padding: Union[bool, str] = True
        max_length: Optional[int] = None
        max_length_labels: Optional[int] = None
        pad_to_multiple_of: Optional[int] = None
        pad_to_multiple_of_labels: Optional[int] = None
        #
        def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]:
            # split inputs and labels since they have to be of different lenghts and need
            # different padding methods
            input_features = [{"input_values": feature["input_values"]} for feature in features]
            label_features = [{"input_ids": feature["labels"]} for feature in features]
            #
            batch = self.processor.pad(
                input_features,
                padding=self.padding,
                max_length=self.max_length,
                pad_to_multiple_of=self.pad_to_multiple_of,
                return_tensors="pt",
            )
            with self.processor.as_target_processor():
                labels_batch = self.processor.pad(
                    label_features,
                    padding=self.padding,
                    max_length=self.max_length_labels,
                    pad_to_multiple_of=self.pad_to_multiple_of_labels,
                    return_tensors="pt",
                )
            #
            # replace padding with -100 to ignore loss correctly
            labels = labels_batch["input_ids"].masked_fill(labels_batch.attention_mask.ne(1), -100)
            #
            batch["labels"] = labels
            #
            return batch
    
    def prepare_dataset(batch):
        inputseq = batch["input_values"]
        #
        # batched output is "un-batched" to ensure mapping is correct
        batch["input_values"] = processor(inputseq, sampling_rate=5000).input_values[0]
        #
        with processor.as_target_processor():
            batch["labels"] = processor("".join(batch["labels"])).input_ids
        return batch

    def compute_metrics(pred):
        pred_logits = pred.predictions
        pred_ids = np.argmax(pred_logits, axis=-1)
        #
        pred.label_ids[pred.label_ids == -100] = processor.tokenizer.pad_token_id
        #
        pred_str = processor.batch_decode(pred_ids)
        # we do not want to group tokens when computing the metrics
        label_str = processor.batch_decode(pred.label_ids, group_tokens=False)
        #
        wer = wer_metric.compute(predictions=pred_str, references=label_str)
        #
        return {"wer": wer}


    sampling_rate = 5_000
    vocab_list = ["[PAD]",'A', 'B', 'C', 'D', 'E', 'F', 'G']

    vocab_dict = {v: k for k, v in enumerate(vocab_list)}
    rev_vocab_dict = {k:v for k, v in enumerate(vocab_list)}
    print(len(vocab_dict))

    with open('note_vocab.json', 'w') as vocab_file:
        json.dump(vocab_dict, vocab_file)

    feature_extractor = Wav2Vec2FeatureExtractor(feature_size=1, sampling_rate=sampling_rate, padding_value=0.0,
                                                 do_normalize=True, return_attention_mask=False)
    tokenizer = Wav2Vec2CTCTokenizer("note_vocab.json", unk_token="[UNK]", pad_token="[PAD]")
    print(tokenizer.pad_token_id)

    processor = Wav2Vec2Processor(feature_extractor=feature_extractor, tokenizer=tokenizer)

    data_collator = DataCollatorCTCWithPadding(processor=processor, padding=True)
    wer_metric = load_metric("wer")
    xdata = np.random.rand(20,4800)
    ydata = np.random.randint(1,len(vocab_list),(20,400))

    df=pd.DataFrame.from_dict({'input_values':[xdata[i] for i,x in enumerate(xdata)],'labels':[[rev_vocab_dict[x] for x in ydata[i]] for i,y in enumerate(ydata)]},orient='columns')
    worddataloader = Dataset.from_dict(df)

    worddataloader = worddataloader.map(prepare_dataset, num_proc=4)

    config = Wav2Vec2Config(vocab_size=tokenizer.vocab_size, 
                            hidden_size=144,
                            m_hidden_layers = 12, num_attention_heads = 12,
                            pad_token_id=tokenizer.pad_token_id,
                            ctc_loss_reduction='sum', mask_time_length=2, mask_feature_length=3,
                            conv_dim=(10, 6),conv_kernel=(3, 3), conv_stride=(2, 2),
                            tdnn_dim = (8, 8 , 8, 7, 88))
    model = Wav2Vec2ForCTC(config=config)

    timestr = time.strftime("%Y%m%d-%H%M%S")
    reponame='model_'+timestr
    training_args = TrainingArguments(
        output_dir=reponame,
        group_by_length=True,
        per_device_train_batch_size=8,
        evaluation_strategy="steps",
        num_train_epochs=45,
        fp16=False,
        gradient_checkpointing=True,
        save_steps=500,
        eval_steps=500,
        logging_steps=1,
        learning_rate=1e-4,
        weight_decay=0.005,
        warmup_steps=1000,
        save_total_limit=3,
    )

    trainer = Trainer(
        model=model,
        data_collator=data_collator,
        args=training_args,
        compute_metrics=compute_metrics,
        train_dataset=worddataloader,
        eval_dataset=worddataloader,
        tokenizer=processor.feature_extractor,
    )
    trainer.train()

if __name__ == '__main__':
    main()
```

Based on print statements, it doesn't seems to break here:
    trainer = Trainer(
        model=model,
        data_collator=data_collator,
        args=training_args,
        compute_metrics=compute_metrics,
        train_dataset=worddataloader,
        eval_dataset=worddataloader,
        tokenizer=processor.feature_extractor,
    )
But what gets very confusing is I can run the demo here without problems. It just on my code and I can't figure out why. https://huggingface.co/blog/fine-tune-xlsr-wav2vec2

Comments:
Hey @dberma15,

What is `bitfusion`? do you have to use it? I'm not familiar with this command

@patrickvonplaten Bitfusion is for using a GPU on a shared resource. Using it is equivalent to saying you want to use GPU resources. If you simply type python script.py, you do not use the GPU resources.


Why not use PyTorch's DDP? E.g.: https://github.com/huggingface/transformers/tree/main/examples/pytorch#distributed-training-and-mixed-precision

Or is this a different use case here?

I was simply following the demo in https://huggingface.co/blog/fine-tune-xlsr-wav2vec2

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

This issue is most likely because the last version of bitfusion (currently version 4.5) only supports PyTorch up to version 1.8. 
See [here ](https://docs.vmware.com/en/VMware-vSphere-Bitfusion/4.5/rn/vmware-vsphere-bitfusion-45-release-notes/index.html) for more. Although PyTorch 1.8.2 is no longer supported it can still be installed, see [here](https://pytorch.org/get-started/previous-versions/#v182-with-lts-support). The only downside of using such an old version is that you might get into trouble when trying to use more than one GPU (at least that's what I experienced). If you're lucky VMware might support newer PyTorch versions in the future but up to now that's the only thing you can do. Also be careful with the CUDA version you're using. Bitfusion 4.5 only supports CUDA up to 11.4.4.
Good luck.