[Trainer] RandomSampler for train dataloader fails with Llama-3.2-1B using max model sequence length
Author: AbrahamSanders
Pull request: no
Labels: name: Ex: LM (Finetuning)

name: trainer

name: bug

### System Info

- `transformers` version: 4.45.2
- Platform: Linux-5.15.0-91-generic-x86_64-with-glibc2.35
- Python version: 3.9.16
- Huggingface_hub version: 0.24.0
- Safetensors version: 0.4.5
- Accelerate version: 1.0.1
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: NO
        - mixed_precision: no
        - use_cpu: False
        - debug: False
        - num_processes: 1
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: 0
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - enable_cpu_affinity: True
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
        - dynamo_config: {'dynamo_backend': 'INDUCTOR'}
- PyTorch version (GPU?): 2.5.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: NVIDIA RTX A6000

### Who can help?

@muellerzr @SunMarc

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run the `run_clm.py` script to finetune Llama-3.2-1B on wikitext-2-raw-v1* as such (single GPU):
```bash
python run_clm.py \
    --log_level info \
    --model_name_or_path=meta-llama/Llama-3.2-1B \
    --dataset_name=Salesforce/wikitext \
    --dataset_config_name=wikitext-2-raw-v1 \
    --block_size=1024 \
    --per_device_train_batch_size=8 \
    --do_train \
    --output_dir=Llama-3.2-1B-wikitext-2-raw-v1 \
    --overwrite_output_dir \
    --seed=42 \
    --logging_steps=10 \
    --lr_scheduler_type=cosine \
    --num_train_epochs=3 \
    --learning_rate=5e-05 \
    --warmup_ratio=0.03 \
    --dataloader_drop_last
```

This works fine.

Now simply remove the `--block_size=1024` arg to let it default to model max length (131072), and re-run.

This produces the following error:
```text
Traceback (most recent call last):
  File "run_clm.py", line 657, in <module>
    main()
  File "run_clm.py", line 605, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/anaconda3/envs/dev/lib/python3.9/site-packages/transformers/trainer.py", line 2052, in train
    return inner_training_loop(
  File "/anaconda3/envs/dev/lib/python3.9/site-packages/transformers/trainer.py", line 2081, in _inner_training_loop
    train_dataloader = self.get_train_dataloader()
  File "/anaconda3/envs/dev/lib/python3.9/site-packages/transformers/trainer.py", line 925, in get_train_dataloader
    dataloader_params["sampler"] = self._get_train_sampler()
  File "/anaconda3/envs/dev/lib/python3.9/site-packages/transformers/trainer.py", line 895, in _get_train_sampler
    return RandomSampler(self.train_dataset)
  File "/anaconda3/envs/dev/lib/python3.9/site-packages/torch/utils/data/sampler.py", line 164, in __init__
    raise ValueError(
**ValueError: num_samples should be a positive integer value, but got num_samples=0**
```

*Note: Dataset size should not be the issue - you can also reproduce this using `--dataset_config_name=wikitext-103-raw-v1`

### Expected behavior

Changing the training sequence length should not affect the operation of the training dataloader.

Comments:
[This](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_clm.py#L519) sets `total_length` to 0 if you are not specifying the `block_size` (it becomes `tokenizer.model_max_length`). This means it has the same behavior if `block_size > len(concatenated_examples[list(examples.keys())[0]])`. Alternative would be to give back the `concatenated_examples` in such cases. The comment already says "you can customize this part to your needs", but maybe there is a good default - let me see if I can propose something...

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.