ZeroDivisionError: integer division or modulo by zero
Author: mingmin95
Pull request: no

### System Info

- `transformers` version: 4.21.1
- Platform: Linux-4.15.0-42-shopee-generic-x86_64-with-glibc2.23
- Python version: 3.9.7
- Huggingface_hub version: 0.2.1
- PyTorch version (GPU?): 1.12.1+cu102 (True)
- Tensorflow version (GPU?): 2.7.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes

### Who can help?

@NielsRogge 

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I used the scripts (https://github.com/huggingface/transformers/blob/main/examples/pytorch/image-pretraining/run_mim.py) to further pretraining swintransformer in my dataset.
I just modified the way to read the data like below
```
ds = {}
if 'train' in data_args.data_files.keys():
    train_images = os.listdir(data_args.data_files['train'])
    train_images_files = [os.path.join(data_args.data_files['train'], image) for image in train_images]
    ds['train'] = Dataset.from_dict({'image': train_images_files}).cast_column("image", Image())
if 'validation' in data_args.data_files.keys():
    val_images = os.listdir(data_args.data_files['validation'])
    val_images_files = [os.path.join(data_args.data_files['validation'], image) for image in val_images]
    ds['validation'] = Dataset.from_dict({'image': val_images_files}).cast_column("image", Image())
```
and the scripts is 
```
python /data/min.ming/project_user_gender_optimization/main/model_further_pretrain.py \
--model_name_or_path /data/min.ming/project_user_gender_optimization/res/swin-tiny-patch4-window7-224 \
--train_dir /data/min.ming/project_user_gender_optimization/data/avatar/train/ID_TRAIN_100000 \
--do_train \
--per_device_train_batch_size 8 \
--num_train_epochs 5 \
--output_dir /data/min.ming/project_user_gender_optimization/tmp/swintransformer_test \
--overwrite_output_dir \
--report_to none
```

but get ZeroDivisionError: integer division or modulo by zero

### Expected behavior

Do you know where is the problem





Comments:
Full error log
```
  File "/data/min.ming/project_user_gender_optimization/main/model_further_pretrain.py", line 437, in <module>
    main()
  File "/data/min.ming/project_user_gender_optimization/main/model_further_pretrain.py", line 411, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/data/min.ming/.conda/envs/mm_py39/lib/python3.9/site-packages/transformers/trainer.py", line 1498, in train
    return inner_training_loop(
  File "/data/min.ming/.conda/envs/mm_py39/lib/python3.9/site-packages/transformers/trainer.py", line 1714, in _inner_training_loop
    for step, inputs in enumerate(epoch_iterator):
  File "/data/min.ming/.conda/envs/mm_py39/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 681, in __next__
    data = self._next_data()
  File "/data/min.ming/.conda/envs/mm_py39/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 721, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/data/min.ming/.conda/envs/mm_py39/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/data/min.ming/.conda/envs/mm_py39/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/data/min.ming/.conda/envs/mm_py39/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 2165, in __getitem__
    return self._getitem(
  File "/data/min.ming/.conda/envs/mm_py39/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 2149, in _getitem
    pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not None else None)
  File "/data/min.ming/.conda/envs/mm_py39/lib/python3.9/site-packages/datasets/formatting/formatting.py", line 491, in query_table
    pa_subtable = _query_table_with_indices_mapping(table, key, indices=indices)
  File "/data/min.ming/.conda/envs/mm_py39/lib/python3.9/site-packages/datasets/formatting/formatting.py", line 57, in _query_table_with_indices_mapping
    return _query_table(table, key)
  File "/data/min.ming/.conda/envs/mm_py39/lib/python3.9/site-packages/datasets/formatting/formatting.py", line 81, in _query_table
    return table.fast_slice(key % table.num_rows, 1)
ZeroDivisionError: integer division or modulo by zero
```

@mm1352363 how did you solved the problem? I'm facing the same issue.

I got the exact same error when I wrapped the model in `model = torch.nn.DataParallel(model)`, passed the wrapped model to `Trainer` and called `trainer.evaluate()` or `trainer.train()`. Turned out the `Trainer` handles multiple GPUs automatically and wrapping the model in `DataParallel` is not necessary. Removing `model = torch.nn.DataParallel(model)` solved the issue.