Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
Author: rajgar114
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.11.3
- Platform: Linux-5.4.0-84-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyTorch version (GPU?): 1.8.1+cu111 (True)
- Tensorflow version (GPU?): 2.5.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.3.4 (cpu)
- Jax version: 0.2.20
- JaxLib version: 0.1.71
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes (Multi GPU setting)


### Who can help
@patrickvonplaten , @patil-suraj 

## Information

Model I am using LED:

The problem arises when using:

the official example scripts
[Finetuneing Longformer Encoder-Decoder (LED)](https://colab.research.google.com/drive/12INTTR6n64TzS4RrXZxMSXfrOd9Xzamo?usp=sharing
)

## To reproduce

Steps to reproduce the behavior:

1. Execute the above script on 2 gpu setting.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
```
../lib/python3.8/site-packages/torch/nn/parallel/_functions.py:65: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all 
```
It seems to be a warning, but the program stops after that & doesn't execute further.
## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
The program should complete the execution of epochs.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hey @rajgar114,

Thanks for the issue! The above script is quite old and we don't actively maintain such scripts. However, I'm happy trying to debug the problem! As a start could you write a minimal executable code example that reproduces the error warning using the most up to date version of `transformers`? E.g. it's probably just the part that includes the `generate()` method that is responsible for the warning / error

Hi @patrickvonplaten,

Here is the minimum executable code that produces the issue:

```python
from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments
from transformers import AutoModelForSeq2SeqLM
from transformers import AutoTokenizer
from datasets import load_dataset
from datasets import load_metric
from datasets import Dataset
import pandas as pd
import datasets


dataList = [['How are you ?', 'Tum kaise ho ?'],
            ['I am fine.', 'Main theek hoon']]

trainDataFrame = pd.DataFrame(dataList, columns = ['English', 'Hinglish'])
print(trainDataFrame.head(2))

trainDataset = Dataset.from_pandas(trainDataFrame[["English", "Hinglish"]])
modelCheckpoint = 'allenai/led-base-16384'
tokenizer = AutoTokenizer.from_pretrained(modelCheckpoint)

maxInputLength = 1024
maxOutputLength = 1024
batchSize = 2

def processDataToModelInputs(batch):
        
    inputs = tokenizer(
        batch['English'],
        padding = 'max_length',
        truncation = True,
        max_length = maxInputLength,
    )
    
    outputs = tokenizer(
        batch['Hinglish'],
        padding = 'max_length',
        truncation = True,
        max_length = maxOutputLength,
    )

    batch["input_ids"] = inputs.input_ids
    batch["attention_mask"] = inputs.attention_mask

    #Create 0 global_attention_mask lists
    batch["global_attention_mask"] = len(batch["input_ids"]) * [
        [0 for _ in range(len(batch["input_ids"][0]))]
    ]

    #Since above lists are references, the following line changes the 0 index for all samples
    batch["global_attention_mask"][0][0] = 1
    batch["labels"] = outputs.input_ids

    # We have to make sure that the PAD token is ignored
    batch["labels"] = [
        [-100 if token == tokenizer.pad_token_id else token for token in labels]
        for labels in batch["labels"]
    ]

    return batch

trainDataset = trainDataset.map(
    processDataToModelInputs,
    batched = True,
    batch_size = batchSize,
    remove_columns = ['English', 'Hinglish']
)
print(trainDataset)

trainDataset.set_format(
    type = "torch",
    columns = ["input_ids", "attention_mask", "global_attention_mask", "labels"]
)

led = AutoModelForSeq2SeqLM.from_pretrained(modelCheckpoint, gradient_checkpointing = False, use_cache = False)

led.config.num_beams = 3
led.config.max_length = 1024
led.config.min_length = 2
led.config.length_penalty = 2.0
led.config.early_stopping = True
led.config.no_repeat_ngram_size = 3

rouge = load_metric("rouge")

def computeMetrics(pred):
    
    #Exact Count Matches
    labels_ids = pred.label_ids
    pred_ids = pred.predictions

    pred_str = tokenizer.batch_decode(pred_ids, skip_special_tokens = True)
    labels_ids[labels_ids == -100] = tokenizer.pad_token_id
    label_str = tokenizer.batch_decode(labels_ids, skip_special_tokens = True)
    
    rouge_output = rouge.compute(
        predictions = pred_str, references = label_str, rouge_types = ["rouge2"]
    )["rouge2"].mid  
    
            
    return {
        "rouge2_precision": round(rouge_output.precision, 4),
        "rouge2_recall": round(rouge_output.recall, 4),
        "rouge2_fmeasure": round(rouge_output.fmeasure, 4)
    }


checkPointingPath = "/home/....."

trainingArgs = Seq2SeqTrainingArguments(
    output_dir =  checkPointingPath + "Outputs/",
    overwrite_output_dir = True, 
    do_train  = True,
    per_device_train_batch_size = batchSize,
    gradient_accumulation_steps = 2,
    num_train_epochs = 2,
    logging_strategy = "epoch",
    save_strategy = "epoch",
    fp16 = True,
    disable_tqdm = False,
    predict_with_generate = True,
    report_to = "none",
    )

trainer = Seq2SeqTrainer(
    model = led,
    tokenizer = tokenizer,
    args = trainingArgs,
    compute_metrics = computeMetrics,
    train_dataset = trainDataset,
)

finalOutput = trainer.train()
file = open(checkPointingPath + 'finalOutput.txt', 'w')
file.write(str(finalOutput))
file.close()
```

I feel may be it's linked to:
[Returning of scalars in Multi-GPU Data Parallel Setting.](https://discuss.pytorch.org/t/how-to-fix-gathering-dim-0-warning-in-multi-gpu-dataparallel-setting/41733)

Hey @rajgar114,

I just checked and your example works fine for me on a single GPU. For multi-GPU I think one has to make use of DDP instead of DP which is also what is recommend by PyTorch now (see first warning here: https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html#torch.nn.DataParallel) 


To continue, could you maybe do the following: 

a) Verify that the code works with single-GPU (set `CUDA_VISIBLE_DEVICES="0"`).
b) Try to run the script with DDP. The Trainer supports DDP out-of-the-box. See: https://github.com/huggingface/transformers/tree/master/examples/pytorch#distributed-training-and-mixed-precision

Let me know if you still run into problems :-)

Hi @patrickvonplaten,

a) I have used the following option to use only the single-GPU as suggested by you:
`os.environ["CUDA_VISIBLE_DEVICES"]="0"`

Using this, the code was working fine and the model was fine-tuned successfully.

b) And for the DDP setting, don't we have a easier way like to pass any argument or something in the Trainer API so that  I can able to directly execute above code in jupyter notebook only without the command line stuff.


for b) DDP sadly relies on Python's multiprocessing so that the script has to be launched by `n` processes. So those are the best docs we have: https://github.com/huggingface/transformers/tree/master/examples/pytorch#distributed-training-and-mixed-precision

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Sorry, could anyone clarify what should be done when using the trainer?
I have the same problem as [rajgar114](https://github.com/rajgar114), but I am using the Huggingface Trainer and the program stops and does not execute further.

`torchrun --nproc_per_node number_of_gpu_you_have path_to_script.py` where `path_to_script.py` is the same Trainer script you were running directly.

Maybe it is related to your `processDataToModelInputs` function. Try to set `return_tensors='pt'` and `padding='max_length'` in the tokenizers. 

I got the same error on Kaggle, but the program still works with this warning. How can I resolve such warnings ?