Why is data_seed not in use
Author: ajesujoba
Pull request: no

### System Info

Is there a use for the  [data_seed](https://github.com/huggingface/transformers/blob/main/src/transformers/training_args.py#L373C9-L373C18) parameter? Why was [data_seed](https://github.com/huggingface/transformers/blob/main/src/transformers/training_args.py#L2944) initialized with [sampler_seed](https://github.com/huggingface/transformers/blob/main/src/transformers/training_args.py#L2890)? Is there any difference between these two arguments? When using run_mlm.py with both parameters set to 32, the results are the same as when they are set to 42. Why is that?




### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Ran run_mlm.py on some monolingual texts for different languages 

### Expected behavior

Expected different model behavior and parameters. 

Comments:
cc @muellerzr 

Hi @ajesujoba , according to my understanding, in the `training_args` script, `data_seed` is the class variable and `sampler_seed` is the kwarg for the `set_dataloader` instance method of `TrainingArguments`  class.  `set_dataloader` is used to give values for dataloading such as `per_device_train_batch_size`, `ignore_data_skip`,...and `data_seed` is one among them for setting a random seed to use in data sampling.These three are the class variables. User will create an instance of `TrainingArguments` and access `set_dataloader` method for providing the necessary parameters for dataloading. And  after doing such and some other things necessary the `TrainingArguments` instance will be passed to `Trainer` instance.
For ex:
```
>>> from transformers import TrainingArguments
>>> training_args =  TrainingArguments("working_dir")
>>> training_args = training_args.set_dataloader(train_batch_size=16, ignore_data_skip=False, sampler_seed =123)
>>> training_args.data_seed
123
```
Cheers!

Thank you very much for your quick response @RUFFY-369. I see your point.  But why is there no randomization when I set the sampler_seed to 32 in [run_mlm.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm.py)? Why is the resulting model the same regardless of the data_seed? Is it due to the kind of [Sampler](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L855C20-L855C40) i.e. `LengthGroupedSampler`?  I also tried following your example above

```training_args = training_args.set_dataloader(train_batch_size=16, ignore_data_skip=False, sampler_seed =123)``` 
within run_mlm. it resulted in the following error:
```
ValueError: prefetch_factor option could only be specified in multiprocessing.let num_workers > 0 to enable multiprocessing.
```
Where and hows is the trainer using the `sampler_seed` or `data_seed`?

Found another previous issue that asked similar question as mine [here](https://github.com/huggingface/transformers/issues/26218). Looking forward to any possible guidiance

@ajesujoba How do you set `sampler_seed` to 32 in [run_ml.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm.py) as the only seed used in this file is training seed, which is the class variable `seed` from `TrainingArguments`:
https://github.com/huggingface/transformers/blob/7729b7747872eabe461a8d2d4ce1068ae0e716a8/examples/pytorch/language-modeling/run_mlm.py#L308.
`seed` is different from `data_seed` or `sampler_seed` as this for model training reproducibility and the other two are for data sampling. set_seed in the `run_mlm` file already sets seed for torch, tensorflow and numpy.
When `data_seed` is not set, the random generators use `seed` value from `TrainingArguments` for data sampling. In `run_mlm` file `set_seed` is used and also in the `Trainer` class for training `set_seed` is [used](https://github.com/huggingface/transformers/blob/9ef93fccade1f2b6349c2eec2188d0ea54329f84/src/transformers/trainer.py#L1870)

Are you changing the `set_seed` arg value in the `run_mlm` file as that determines reproducibility of the trained model? If you want to change the result of trained model then try setting different `seed` for `training_args` before model instantiation like this in the `run_mlm` file: `training_args = training_args.set_training(seed=55)` after `training_args` is defined
 


>  I also tried following your example above
> 
> `training_args = training_args.set_dataloader(train_batch_size=16, ignore_data_skip=False, sampler_seed =123)` within run_mlm. it resulted in the following error:
> 
> ```
> ValueError: prefetch_factor option could only be specified in multiprocessing.let num_workers > 0 to enable multiprocessing.
> ```
This example was a dummy example for explanation purpose and not for the actual code 

> 
> Where and hows is the trainer using the `sampler_seed` or `data_seed`?
> 

And I don't think trainer file is using the `sampler_seed` or `data_seed` as `set_seed` is used for data sampling as well as training 





@RUFFY-369 , thanks for your response. I set data_seed/sampler_seed by passing the argument `--data_seed 32` in the training command `python run_mlm.py --data_seed 32 --seed 42`. I totally understand that `seed` and `data_seed` are different arguments. And for my experiment I set both `--data_seed` and `seed` to different values .e.g.32 and 42 respectively. But the training dynamics is always giving the same result as when `-data_seed` is not set.

Can you point me to where the `data_seed` parameter is used in the codebase for sampling or where it is passed to the sampler? For seed, it is `set_seed()` both within the trainer and the run_mlm.py. 

> And I don't think trainer file is using the `sampler_seed` or `data_seed` as `set_seed` is used for data sampling as well as training
>
.
> If you want to change the result of trained model then try setting different `seed` for `training_args` before model instantiation like this in the `run_mlm` file: `training_args = training_args.set_training(seed=55)` after `training_args` is defined
> 
Please give `seed` a value other than 42 as it is already the default value in `training_args`
https://github.com/huggingface/transformers/blob/14ff5dd962c1bd0a4e3adaac347ba396d8df5add/src/transformers/training_args.py#L998

Then I guess it is safe to say that `data_seed` is an unused variable and the meta data 
```Random seed to be used with data samplers. If not set, random generators for data sampling will use the same seed as `seed`.```
provided does not hold. Only `seed` is useful. 

Sorry for digging up a closed issue, but I think the `data_seed` argument is no longer used. Searching for `data_seed` in the codebase returns [no result](https://github.com/search?q=repo%3Ahuggingface%2Ftransformers%20data_seed&type=code) other than the declaration in `training_args.py`. It seems that it was originally used in `Trainer. _get_train_sampler()`, but this is no longer the case.

@amyeroberts Should we deprecate this flag, as it has no effect, and amend the docs?

cc @muellerzr for advice