[Flax][BART Example] Always do map to generate cache files each run when running examples/flax/language-modeling/run_bart_dlm_flax.py
Author: jameszhouyi
Pull request: no

### System Info

Hi, 
I am running Flax BRAT example model on Ubuntun 22.04 with run_bart_dlm_flax.py(commit ID: 0afa5071bd84e44301750fdc594e33db102cf374).  But I found it always do Map like below before each running training model even if there are already cache* file in local disk. It causes lots of caches file will occupy my disk space due to increasing cache files. Do you please help me to check this issue ? thanks in advanced. 

Map: 100%| 3068443/3068443 [14:21<00:00, 3561.40 examples/s]
Map: 100%| 161497/161497 [00:44<00:00, 3641.38 examples/s]
Map:  11%| 338000/3068443 [01:29<13:25, 3388.28 examples/s

-rw-r--r-- 1 root root  5062419936 Sep 21 10:49 cache-d06d4112eceed0a0.arrow
-rw-r--r-- 1 root root   265334832 Sep 21 10:50 cache-e28e13db394c2fd6.arrow
-rw-r--r-- 1 root root 13110709912 Sep 21 11:05 cache-bf0c9463974edb34.arrow
-rw-r--r-- 1 root root   687149936 Sep 21 11:06 cache-101c2523a1342475.arrow



### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

 Running examples/flax/language-modeling/run_bart_dlm_flax.py for reproducing.

### Expected behavior

if there are already cache files in local disk, it will not do again map when running the model example with run_bart_dlm_flax.py.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi,

Could anyone please have a look this issue ? Thanks

Hey! Sorry we might have missed this one. I think this is expected given that you have to process examples that you don't have. Once they are processed once I expected them to be cached. You need enough space for the full dataset you are using 

hi @ArthurZucker 
Thanks for your response. This issue is that the more cache files will be stored in local disk, the more test case run repeatedly.  This will lead to occupy a lot of local disk space :-( . BTW, in other test case(e.g.,examples/flax/language-modeling/run_clm_flax.py ), there is no new cache files to be generated if the cache files has been existed. I am not sure why the case of run_bart_dlm_flax.py has different behavior. Please help me have a look this issue. Thanks in advance.

Thanks for checking that. I'll ping @sanchit-gandhi if he has time! 🤗 

Hey @jameszhouyi - are you setting the flag `overwrite_cache=True`? Could you make sure you're setting this to `False` so that we load from cache:
https://github.com/huggingface/transformers/blob/c832bcb812fc962830c11ea64c5ff623240a3d6d/examples/flax/language-modeling/run_clm_flax.py#L578

Hi @sanchit-gandhi , thanks for your support!

I have set load_from_cache_file=True like below code snippet in https://github.com/huggingface/transformers/blob/main/examples/flax/language-modeling/run_bart_dlm_flax.py#L701C15 

    tokenized_datasets = tokenized_datasets.map(
        group_texts,
        batched=True,
        num_proc=data_args.preprocessing_num_workers,
        load_from_cache_file=True,
    )

But i found run the BART model and it still do mapping and generate the cache files (e.g., cache-697b211c6e710ef2.arrow) even if above setting:
python run_bart_dlm_flax.py \
>     --output_dir="./norwegian-bart-base" \
>     --model_type="bart" \
>     --config_name="./norwegian-bart-base" \
>     --tokenizer_name="./norwegian-bart-base" \
>     --dataset_name="/mnt/disk1/hf_datasets/oscar" \
>     --do_train \
>     --dataset_config_name="unshuffled_deduplicated_no" \
>     --max_seq_length="1024" \
>     --per_device_train_batch_size="16" \
>     --per_device_eval_batch_size="16" \
>     --learning_rate="1e-4" \
>     --warmup_steps="5" \
>     --overwrite_output_dir \
>     --num_train_epochs="3" \
>     --logging_steps="500" \
>     --save_steps="2000" \
>     --eval_steps="2000" \
>     --dtype="bfloat16" \
>     --num_iterations="100" \
>     --cache_dir="/mnt/disk1/hf_datasets/oscar_bart"
[14:39:34] - INFO - __main__ - Training/evaluation parameters TrainingArguments(output_dir='./norwegian-bart-base', overwrite_output_dir=True, do_train=True, do_eval=False, per_device_train_batch_size=16, per_device_eval_batch_size=16, learning_rate=0.0001, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, adafactor=False, num_train_epochs=3.0, warmup_steps=5, logging_steps=500, save_steps=2000, eval_steps=2000, seed=42, push_to_hub=False, hub_model_id=None, hub_token=None, num_iterations=100)
/mnt/disk1/miniconda3/envs/test/lib/python3.10/site-packages/datasets/table.py:1421: FutureWarning: promote has been superseded by mode='default'.
  table = cls._concat_blocks(blocks, axis=0)
[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
Map:  70%|████████████████████████████████████████████████▉                     | 2145625/3068443 [09:37<04:02, 3800.19 examples/s

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.