Bug: RuntimeError: Tensors must be contiguous
Author: yorhaha
Pull request: no

### System Info

- `transformers` version: 4.31.0
- Platform: Linux-5.4.0-146-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.2
- Accelerate version: 0.21.0
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: DEEPSPEED
        - mixed_precision: bf16
        - use_cpu: False
        - num_processes: 4
        - machine_rank: 0
        - num_machines: 1
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - deepspeed_config: {'gradient_accumulation_steps': 4, 'gradient_clipping': 0.5, 'offload_optimizer_device': 'none', 'offload_param_device': 'none', 'zero3_init_flag': False, 'zero_stage': 2}
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@pacman

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I used my own dataset to train with a customized trainer class (inherited from `transformers.Seq2SeqTrainer`). It reported error `RuntimeError: Tensors must be contiguous`.

If I change the `_gpu_gather` function of `/venv/lib/python3.8/site-packages/accelerate/utils/operations.py` as following, it becomes well:

```
def _gpu_gather(tensor):
    def _gpu_gather_one(tensor):
        if tensor.ndim == 0:
            tensor = tensor.clone()[None]

        ################################################### make tensors to be contiguous
        output_tensors = [torch.empty_like(tensor).contiguous() for _ in range(torch.distributed.get_world_size())]
        torch.distributed.all_gather(output_tensors, tensor.contiguous())
        ###################################################

        return torch.cat(output_tensors, dim=0)

    return recursively_apply(_gpu_gather_one, tensor, error_on_other_type=True)
```

### Expected behavior

```
  File "/venv/lib/python3.8/site-packages/transformers/trainer.py", line 3147, in evaluation_loop
    logits = self.accelerator.gather_for_metrics((logits))
  File "/venv/lib/python3.8/site-packages/accelerate/accelerator.py", line 2012, in gather_for_metrics
    tensor = self.gather(tensor)
  File "/venv/lib/python3.8/site-packages/accelerate/accelerator.py", line 1985, in gather
    return gather(tensor)
  File "/venv/lib/python3.8/site-packages/accelerate/utils/operations.py", line 289, in gather
    return _gpu_gather(tensor)
  File "/venv/lib/python3.8/site-packages/accelerate/utils/operations.py", line 269, in _gpu_gather
    return recursively_apply(_gpu_gather_one, tensor, error_on_other_type=True)
  File "/venv/lib/python3.8/site-packages/accelerate/utils/operations.py", line 109, in recursively_apply
    return honor_type(
  File "/venv/lib/python3.8/site-packages/accelerate/utils/operations.py", line 83, in honor_type
    return type(obj)(generator)
  File "/venv/lib/python3.8/site-packages/accelerate/utils/operations.py", line 112, in <genexpr>
    recursively_apply(
  File "/venv/lib/python3.8/site-packages/accelerate/utils/operations.py", line 128, in recursively_apply
    return func(data, *args, **kwargs)
  File "/venv/lib/python3.8/site-packages/accelerate/utils/operations.py", line 266, in _gpu_gather_one
    torch.distributed.all_gather(output_tensors, tensor)
  File "/venv/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1451, in wrapper
    return func(*args, **kwargs)
  File "/venv/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 2448, in all_gather
    work = default_pg.allgather([tensor_list], [tensor])
RuntimeError: Tensors must be contiguous
```

Comments:
This has been fixed in Accelerate, please install that library from source.