RuntimeError: Tensors must be contiguous error while finetuning with deepspeed.
Author: FahriBilici
Pull request: no

### System Info

- `transformers` version: 4.25.1
- Platform: Linux-5.10.147+-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.13.1+cu116 (True)
- Tensorflow version (GPU?): 2.9.2 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


### Who can help?

@stas00 @ArthurZucker @sgugger 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am just trying to fine-tune "EleutherAI/gpt-neo-1.3B" for casuallm on google colab. Without anything, it gives out of memory error. I was checking what can I do and I found deepspeed. I added deepspeed='ds_config.json', to my training arguments in jupyter notebook and used configuration from the official page which is "ds_config_zero2.json". 

### Expected behavior

start training.

Comments:
1.3B param of weights + grads + optim states in mixed precision would need about `18*1.3=24`GB of memory, plus you need more memory for activations and temps and cuda kernels.

The free colab account is too limited to do much on it with even a small model. It barely has any cpu memory so no memory to offload to.

You could use deepspeed to offload to local disk (`nvme`), it'll be slow but doable I think if your local disc is large enough. Please see: https://huggingface.co/docs/transformers/main/main_classes/deepspeed#nvme-support



The other approach is to activate BNB's Adam, so it will cut down on a lot of optim states weights (2 bytes instead of 8) except the embedding params at full 8 bytes. so you will be looking at about 17GB for weights + grads + optim states in mixed precision - but it's still too large for colab without offloading.

my actual goal is finetuning gpt-j on google colab pro but since google colab uses credits I am experimenting with 1.3B on normal colab. I also used nvme settings with zero3 example but still I got the same error without aio part. if I add aio part I got  `ValidationError: 1 validation error for DeepSpeedZeroConfig
aio
  extra fields not permitted (type=value_error.extra)`

Understood.

deepspeed's `nvme` offload requires `libaio`. 

As we only integrate deepspeed, any questions about deepspeed functionality itself and errors such as above should be posted at https://github.com/microsoft/DeepSpeed/issues since we aren't the maintainers of deepspeed.

Thank you.

but still `RuntimeError: Tensors must be contiguous` happens. I saw you made merge about fixing this but that still happens.


I'm struggling here with supporting you, @FahriBilici - please kindly read 
https://github.com/huggingface/transformers/blob/main/ISSUES.md#the-github-issues
and file a proper issue with the full traceback and invocation command, I will be able to help you then.
Thanks.

I will share my colab notebook and training set once I prepare.

I repeat what's needed is the command line and the full traceback. Thank you. 

the full error is 
```
The following columns in the training set don't have a corresponding argument in "GPTNeoForCausalLM.forward" and have been ignored: text. If text are not expected by "GPTNeoForCausalLM.forward",  you can safely ignore this message.
Detected ZeRO Offload and non-DeepSpeed optimizers: This combination should work as long as the custom optimizer has both CPU and GPU implementation (except LAMB)
[2023-01-21 10:17:13,756] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed info: version=0.8.0, git-hash=unknown, git-branch=unknown
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
[<ipython-input-23-3435b262f1ae>](https://localhost:8080/#) in <module>
----> 1 trainer.train()

10 frames
[/usr/local/lib/python3.8/dist-packages/torch/distributed/distributed_c10d.py](https://localhost:8080/#) in broadcast(tensor, src, group, async_op)
   1402         group_src_rank = get_group_rank(group, src)
   1403         opts.rootRank = group_src_rank
-> 1404         work = group.broadcast([tensor], opts)
   1405     if async_op:
   1406         return work

RuntimeError: Tensors must be contiguous
```

my config file is 
```
{
    "zero_optimization": {
        "stage": 3,
        "offload_optimizer": {
            "device": "nvme",
            "nvme_path": "/local_nvme",
            "pin_memory": true,
            "buffer_count": 4,
            "fast_init": false
        },
        "offload_param": {
            "device": "nvme",
            "nvme_path": "/local_nvme",
            "pin_memory": true,
            "buffer_count": 5,
            "buffer_size": 1e8,
            "max_in_cpu": 1e9
        },
        "overlap_comm": true,
        "contiguous_gradients": true,
        "sub_group_size": 1e9,
        "reduce_bucket_size": "auto",
        "stage3_prefetch_bucket_size": "auto",
        "stage3_param_persistence_threshold": "auto",
        "stage3_max_live_parameters": 1e9,
        "stage3_max_reuse_distance": 1e9,
        "stage3_gather_16bit_weights_on_model_save": true

    },
    "train_batch_size": "auto",
    "train_micro_batch_size_per_gpu": "auto"
}
```
my training code is 

```
from transformers import TrainingArguments, Trainer

training_args = TrainingArguments(
    output_dir="neo",
    evaluation_strategy="epoch",
    learning_rate=2e-5,
    num_train_epochs=10,
    weight_decay=0.01,
    gradient_checkpointing=True,
    deepspeed='config.json',
    report_to=None
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_datasets['train'],
    eval_dataset=tokenized_datasets['validation'],
    data_collator=data_collator,
)
trainer.train()

```

ok, clearly we have a miscommunication here. I will try one last time.

To help you we need the **full traceback** and not the last line of it. 

```
The following columns in the training set don't have a corresponding argument in `GPTNeoForCausalLM.forward` and have been ignored: text. If text are not expected by `GPTNeoForCausalLM.forward`,  you can safely ignore this message.
Detected ZeRO Offload and non-DeepSpeed optimizers: This combination should work as long as the custom optimizer has both CPU and GPU implementation (except LAMB)
[2023-01-21 10:17:13,756] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed info: version=0.8.0, git-hash=unknown, git-branch=unknown
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-23-3435b262f1ae> in <module>
----> 1 trainer.train()

10 frames
/usr/local/lib/python3.8/dist-packages/transformers/trainer.py in train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
   1525             self._inner_training_loop, self._train_batch_size, args.auto_find_batch_size
   1526         )
-> 1527         return inner_training_loop(
   1528             args=args,
   1529             resume_from_checkpoint=resume_from_checkpoint,

/usr/local/lib/python3.8/dist-packages/transformers/trainer.py in _inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)
   1594         )
   1595         if args.deepspeed:
-> 1596             deepspeed_engine, optimizer, lr_scheduler = deepspeed_init(
   1597                 self, num_training_steps=max_steps, resume_from_checkpoint=resume_from_checkpoint
   1598             )

/usr/local/lib/python3.8/dist-packages/transformers/deepspeed.py in deepspeed_init(trainer, num_training_steps, resume_from_checkpoint, inference)
    342     )
    343 
--> 344     deepspeed_engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)
    345 
    346     if resume_from_checkpoint is not None:

/usr/local/lib/python3.8/dist-packages/deepspeed/__init__.py in initialize(args, model, optimizer, model_parameters, training_data, lr_scheduler, mpu, dist_init_required, collate_fn, config, config_params)
    123 
    124     if not isinstance(model, PipelineModule):
--> 125         engine = DeepSpeedEngine(args=args,
    126                                  model=model,
    127                                  optimizer=optimizer,

/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/engine.py in __init__(self, args, model, optimizer, model_parameters, training_data, lr_scheduler, mpu, dist_init_required, collate_fn, config, config_params, dont_change_device)
    299 
    300         # Configure distributed model
--> 301         self._configure_distributed_model(model)
    302 
    303         self._get_model_parameters()

/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/engine.py in _configure_distributed_model(self, model)
   1185 
   1186         if not self.amp_enabled():
-> 1187             self._broadcast_model()
   1188 
   1189     # check if parameters are duplicated in optimizer param_groups

/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/engine.py in _broadcast_model(self)
   1100             else:
   1101                 if torch.is_tensor(p) and is_replicated(p):
-> 1102                     dist.broadcast(p,
   1103                                    groups._get_broadcast_src_rank(),
   1104                                    group=self.data_parallel_group)

/usr/local/lib/python3.8/dist-packages/deepspeed/comm/comm.py in log_wrapper(*args, **kwargs)
    125         # Return the op, then stop the op's timer
    126         try:
--> 127             return func(*args, **kwargs)
    128         finally:
    129             if comms_logger.enabled:

/usr/local/lib/python3.8/dist-packages/deepspeed/comm/comm.py in broadcast(tensor, src, group, async_op, prof, log_name, debug)
    230               debug=get_caller_func()):
    231     global cdb
--> 232     return cdb.broadcast(tensor=tensor, src=src, group=group, async_op=async_op)
    233 
    234 

/usr/local/lib/python3.8/dist-packages/deepspeed/comm/torch.py in broadcast(self, tensor, src, group, async_op)
     68 
     69     def broadcast(self, tensor, src, group=None, async_op=False):
---> 70         return torch.distributed.broadcast(tensor=tensor,
     71                                            src=src,
     72                                            group=group,

/usr/local/lib/python3.8/dist-packages/torch/distributed/distributed_c10d.py in broadcast(tensor, src, group, async_op)
   1402         group_src_rank = get_group_rank(group, src)
   1403         opts.rootRank = group_src_rank
-> 1404         work = group.broadcast([tensor], opts)
   1405     if async_op:
   1406         return work

RuntimeError: Tensors must be contiguous
```

Excellent. Thank you for providing the full traceback, @FahriBilici 

As you can see the issue comes from inside deepspeed and is unrelated to the fix I made earlier even though the error message is the same. Therefore you want to report it here https://github.com/microsoft/DeepSpeed/issues

Alternatively, you can traverse your model before you pass it to the Trainer and ensure that all tensors are contiguous.

Probably something along the lines of:
```
for p in model.parameters():
    p = p.contiguous()
```

I haven't tested it, this is just an idea to try.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.