run summarization
Author: tigernandita
Pull request: no

### System Info

with open(output_prediction_file, "w", encoding="utf-8") as writer:: The script opens the output file in write mode with UTF-8 encoding to handle non-ASCII characters

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

with open(output_prediction_file, "w" as writer)
UnicodeEncodeError: 'ascii' codec can't encode character '\xe2' in position 1710: ordinal not in range(128)

### Expected behavior

It should generate predictions

Comments:
@tigernandita 

Please report the issue with providing your env. info.

You can run the command `transformers-cli env` and copy-paste its output.

Also, please provide the full traceback.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@ydshieh  I also got this problem when training LED. Here my env
- `transformers` version: 4.36.0.dev0
- Platform: Linux-5.15.0-88-generic-x86_64-with-glibc2.35
- Python version: 3.11.5
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: 0.23.0
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: NO
        - mixed_precision: fp16
        - use_cpu: False
        - debug: False
        - num_processes: 1
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

Here is the full traceback 
Traceback (most recent call last):
  File "/home/ntq/nhanv/NLP/hf_trainer/NER/run_summarization_2.py", line 788, in <module>
    main()
  File "/home/ntq/nhanv/NLP/hf_trainer/NER/run_summarization_2.py", line 760, in main
    writer.write("\n".join(predictions))
UnicodeEncodeError: 'ascii' codec can't encode character '\u2019' in position 292: ordinal not in range(128)

Hi @huyhuyvu01 

we need a minimal (but self complete) reproducible code snippet to help.

@ydshieh I train LED using the run_summarization.py examples using the below lines
CUDA_VISIBLE_DEVICES=0 python3 run_summarization.py \
    --model_name_or_path allenai/led-base-16384 \
    --train_file train.json \
    --validation_file test.json \
    --test_file test.json \
    --output_dir /hdd-6tb/llm_model/ouputs \
    --do_train \
    --do_predict \
    --num_train_epochs 10 \
    --source_prefix "summarize: " \
    --text_column "prompt" \
    --summary_column "completion" \
    --per_device_train_batch_size=8 \
    --per_device_eval_batch_size=8 \
    --logging_strategy steps\
    --evaluation_strategy epoch \
    --save_strategy epoch \
    --logging_steps 100 \
    --overwrite_output_dir \
    --predict_with_generate
    

Could you provide the json files (even the fake, tiny versions), or replace them with some public datasets on the Hub?