Issue 1:
It seems like google/roberta2roberta_L-24_wikisplit should be pre-processed differently than originally thought:

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("google/roberta2roberta_L-24_wikisplit")
model = AutoModelForSeq2SeqLM.from_pretrained("google/roberta2roberta_L-24_wikisplit")

long_sentence = """Due to the hurricane, Lobsterfest has been canceled, making Bob very happy about it and he decides to open Bob 's Burgers for customers who were planning on going to Lobsterfest."""

input_ids = tokenizer(long_sentence, return_tensors="pt").input_ids
output_ids = model.generate(input_ids)[0]
print(tokenizer.decode(output_ids, skip_special_tokens=True))
# should output
# Due Due hurricane, Lobsterfest has been canceled, making Bob very happy about it. He decides to open B
# ob's Burgers for customers who were planning on going to Lobsterfest.com.

yields a weird word duplication bug, whereas:

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("google/roberta2roberta_L-24_wikisplit")
model = AutoModelForSeq2SeqLM.from_pretrained("google/roberta2roberta_L-24_wikisplit")

long_sentence = """Due to the hurricane, Lobsterfest has been canceled, making Bob very happy about it and he decides to open Bob 's Burgers for customers who were planning on going to Lobsterfest."""

input_ids = tokenizer(tokenizer.bos_token + long_sentence + tokenizer.eos_token, return_tensors="pt").input_ids
output_ids = model.generate(input_ids)[0]
print(tokenizer.decode(output_ids, skip_special_tokens=True))
# should output
# Due to the hurricane, Lobsterfest has been canceled, making Bob very happy about it. He decides to open Bob's Burgers for customers who were planning on going to Lobsterfest.

yields good results.

Notice the difference of

input_ids = tokenizer(tokenizer.bos_token + long_sentence + tokenizer.eos_token, return_tensors="pt").input_ids between code sample 1 and 2.

@patrickvonplaten

Wait for author's (@shashiongithub) answer before changing code example.

Issue 2:
System Info

transformers version: 4.28.0.dev0

Platform: Linux-4.15.0-175-generic-x86_64-with-glibc2.27

Python version: 3.10.10

Huggingface_hub version: 0.13.4

Safetensors version: not installed

PyTorch version (GPU?): 1.12.0 (True)

Tensorflow version (GPU?): not installed (NA)

Flax version (CPU?/GPU?/TPU?): not installed (NA)

Jax version: not installed

JaxLib version: not installed

Using GPU in script?:

Using distributed or parallel set-up in script?:

Who can help?

@sgugger @stas00

Information

The official example scripts

My own modified scripts

Tasks

An officially supported task in the examples folder (such as GLUE/SQuAD, ...)

My own task or dataset (give details below)

Reproduction

I am training GPT on 2 nodes, each with 8 GPUs currently. I used the Trainer API provided by huggingface for training. In addition, I used Deepspeed's ZeRO3 strategy. I have successfully start training, but there are the following issues in the output log:

Due to the presence of two nodes, there will be two rows of output at a time, each from a different node. I observed that the output of two nodes is the same, and the following is an example. Do I only need to focus on one line of output?(in other words, one of these two lines is redundant.) Or do these two lines mean that two machines have fed the same data and print the same output?

node1: {'loss': 1.4406, 'learning_rate': 2e-05, 'epoch': 0.17}
node2: {'loss': 1.4406, 'learning_rate': 2e-05, 'epoch': 0.17}
node1: {'loss': 1.4457, 'learning_rate': 2e-05, 'epoch': 0.18}
node2: {'loss': 1.4457, 'learning_rate': 2e-05, 'epoch': 0.18}

There will be a progress bar during single node training, which should be performed using tqdm. However, there is no progress bar during the training.

In the training command, I used --report_to "tensorboard", but I did not find any output in tensorboard.

Here is my command to start training.

deepspeed --num_gpus 8 --num_nodes 2 --hostfile=host.txt train.py \
--model_name_or_path /path/to/model \
--data_path /path/to/data \
--output_dir /path/to/output \
--num_train_epochs 1 \
--per_device_train_batch_size 32 \
--per_device_eval_batch_size 1 \
--gradient_accumulation_steps 1 \
--evaluation_strategy "no" \
--save_strategy "steps" \
--save_steps 100 \
--save_total_limit 2 \
--learning_rate 2e-5 \
--logging_steps 1 \
--report_to "tensorboard" \
--gradient_checkpointing True \
--deepspeed configs/deepspeed_config.json \
--fp16 True

Expected behavior

I hope you can provide answers to the above three questions. Thanks!