Multiple Node Training Log
Author: MikeDean2367
Pull request: no

### System Info

- `transformers` version: 4.28.0.dev0
- Platform: Linux-4.15.0-175-generic-x86_64-with-glibc2.27
- Python version: 3.10.10
- Huggingface_hub version: 0.13.4
- Safetensors version: not installed
- PyTorch version (GPU?): 1.12.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@sgugger @stas00

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am training GPT on 2 nodes, each with 8 GPUs currently. I used the `Trainer` API provided by huggingface for training. In addition, I used Deepspeed's `ZeRO3` strategy. I have successfully start training, but there are the following issues in the output log:
1. Due to the presence of two nodes, there will be two rows of output at a time, each from a different node. I observed that the output of two nodes is the same, and the following is an example. Do I only need to focus on one line of output?(in other words, one of these two lines is redundant.) Or do these two lines mean that two machines have fed the same data and print the same output?
   ```shell
   node1: {'loss': 1.4406, 'learning_rate': 2e-05, 'epoch': 0.17}
   node2: {'loss': 1.4406, 'learning_rate': 2e-05, 'epoch': 0.17}
   node1: {'loss': 1.4457, 'learning_rate': 2e-05, 'epoch': 0.18}
   node2: {'loss': 1.4457, 'learning_rate': 2e-05, 'epoch': 0.18}
   ```
2. There will be a progress bar during single node training, which should be performed using `tqdm`. However, there is no progress bar during the training.
3. In the training command, I used `--report_to "tensorboard"`, but I did not find any output in `tensorboard`.

Here is my command to start training.
```shell
deepspeed --num_gpus 8 --num_nodes 2  --hostfile=host.txt train.py \
    --model_name_or_path /path/to/model \
    --data_path /path/to/data \
    --output_dir /path/to/output \
    --num_train_epochs 1 \
    --per_device_train_batch_size 32 \
    --per_device_eval_batch_size 1 \
    --gradient_accumulation_steps 1 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 100 \
    --save_total_limit 2 \
    --learning_rate 2e-5 \
    --logging_steps 1 \
    --report_to "tensorboard" \
    --gradient_checkpointing True \
    --deepspeed configs/deepspeed_config.json \
    --fp16 True
```

### Expected behavior

I hope you can provide answers to the above three questions. Thanks!

Comments:
By default `--log_on_each_node` is `True` but you can set it to `False` to avoid the duplicate logs. I don't know if DeepSpeed does anything to the default log levels, the progress bar should be there on the two main nodes by default (every process that has a log level high enough).

> By default `--log_on_each_node` is `True` but you can set it to `False` to avoid the duplicate logs. I don't know if DeepSpeed does anything to the default log levels, the progress bar should be there on the two main nodes by default (every process that has a log level high enough).

Thank you for your reply. Your answer made me want to know the answer to the first question. For the second question, when the training was shuted down, the progress bar appeared. I don't know why. In my experiment, all outputs were only on the master node, and non master nodes had no outputs. For the last question, I think @stas00 can help.

> By default `--log_on_each_node` is `True` but you can set it to `False` to avoid the duplicate logs. I don't know if DeepSpeed does anything to the default log levels, the progress bar should be there on the two main nodes by default (every process that has a log level high enough).

Hi, I tried adding some arguments to the original command to prevent this redundant output. Here are my two ways:
`--log_on_each_node False` `--log_level warning --log_level_replica error --log_on_each_node 0`. It doesn't seem to have much effect because redundant output is still generated on one node.

For the third question, I found the output in `tensorboard`. I tried to change the master node, so I solved the third problem. For the first two problems, I think the first one still needs to be solved urgently because I have tried various solutions but have not been able to solve them. For the second question, it is possible to estimate the training time in Tensorboard, but I still hope to improve this issue. I guess if the first problem is solved, then the second problem can also be solved.

As you closed the issue I'm not sure if there is anything remaining to address here.

The integration code does propagate the log-level setting here:

https://github.com/huggingface/transformers/blob/9858195481e0d29e9b720705d359f98620680a06/src/transformers/deepspeed.py#L350

but the outputs you shared in the OP come from HF Trainer and not Deepspeed.

Thank you for your reply. So does this mean that the first two issues I mentioned are from Huggingface's Trainer and not Deepspeed?

The first log is coming from HF Trainer - if you're not sure what comes from where it's very simple to test. Turn deepspeed off and see what you get as a baseline. If the model is too big, swap in a tiny model - we have one for each arch here: https://huggingface.co/hf-internal-testing

Wrt tqdm the training progress bar is there with deepspeed, at least on a single node. I have just tested it.

But let's first align so that we are testing the same code, please use this example (from `transformers` git clone top level dir)

```
$ PYTHONPATH=src USE_TF=0 deepspeed --num_gpus 2 --num_nodes 1 \
examples/pytorch/translation/run_translation.py --model_name_or_path \
patrickvonplaten/t5-tiny-random --output_dir /tmp/zero3 --overwrite_output_dir \
--max_train_samples 40 --max_eval_samples 40 --max_source_length 128 \
--max_target_length 128 --val_max_target_length 128 --do_train --do_eval \
--num_train_epochs 1 --per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 --learning_rate 3e-3 --warmup_steps 500 \
--predict_with_generate --save_steps 0 --eval_steps 0 --group_by_length \
--dataset_name wmt16 --dataset_config ro-en --source_lang en --target_lang ro \
--source_prefix 'translate English to Romanian: ' --deepspeed \
tests/deepspeed/ds_config_zero3.json --logging_steps 5 --logging_strategy \
steps
```

and check that it logs correctly 

then try with `deepspeed --num_gpus 2 --num_nodes 2` and check again. If something doesn't look right, we will sort it out.

As Sylvain mentioned you will probably need to set `--log_on_each_node 0` in that multi-node experiment.

And to run the same w/o deepspeed, so that you could check the baseline:

```
$ PYTHONPATH=src USE_TF=0 torchrun --nproc-per-node 2 --nnodes 2 \
examples/pytorch/translation/run_translation.py --model_name_or_path \
patrickvonplaten/t5-tiny-random --output_dir /tmp/zero3 --overwrite_output_dir \
--max_train_samples 40 --max_eval_samples 40 --max_source_length 128 \
--max_target_length 128 --val_max_target_length 128 --do_train --do_eval \
--num_train_epochs 1 --per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 --learning_rate 3e-3 --warmup_steps 500 \
--predict_with_generate --save_steps 0 --eval_steps 0 --group_by_length \
--dataset_name wmt16 --dataset_config ro-en --source_lang en --target_lang ro \
--source_prefix 'translate English to Romanian: ' --logging_steps 5 --logging_strategy \
steps
```

> The first log is coming from HF Trainer - if you're not sure what comes from where it's very simple to test. Turn deepspeed off and see what you get as a baseline. If the model is too big, swap in a tiny model - we have one for each arch here: https://huggingface.co/hf-internal-testing
> 
> Wrt tqdm the training progress bar is there with deepspeed, at least on a single node. I have just tested it.
> 
> But let's first align so that we are testing the same code, please use this example (from `transformers` git clone top level dir)
> 
> ```
> $ PYTHONPATH=src USE_TF=0 deepspeed --num_gpus 2 --num_nodes 1 \
> examples/pytorch/translation/run_translation.py --model_name_or_path \
> patrickvonplaten/t5-tiny-random --output_dir /tmp/zero3 --overwrite_output_dir \
> --max_train_samples 40 --max_eval_samples 40 --max_source_length 128 \
> --max_target_length 128 --val_max_target_length 128 --do_train --do_eval \
> --num_train_epochs 1 --per_device_train_batch_size 1 \
> --per_device_eval_batch_size 1 --learning_rate 3e-3 --warmup_steps 500 \
> --predict_with_generate --save_steps 0 --eval_steps 0 --group_by_length \
> --dataset_name wmt16 --dataset_config ro-en --source_lang en --target_lang ro \
> --source_prefix 'translate English to Romanian: ' --deepspeed \
> tests/deepspeed/ds_config_zero3.json --logging_steps 5 --logging_strategy \
> steps
> ```
> 
> and check that it logs correctly
> 
> then try with `deepspeed --num_gpus 2 --num_nodes 2` and check again. If something doesn't look right, we will sort it out.
> 
> As Sylvain mentioned you will probably need to set `--log_on_each_node 0` in that multi-node experiment.
> 
> And to run the same w/o deepspeed, so that you could check the baseline:
> 
> ```
> $ PYTHONPATH=src USE_TF=0 torchrun --nproc-per-node 2 --nnodes 2 \
> examples/pytorch/translation/run_translation.py --model_name_or_path \
> patrickvonplaten/t5-tiny-random --output_dir /tmp/zero3 --overwrite_output_dir \
> --max_train_samples 40 --max_eval_samples 40 --max_source_length 128 \
> --max_target_length 128 --val_max_target_length 128 --do_train --do_eval \
> --num_train_epochs 1 --per_device_train_batch_size 1 \
> --per_device_eval_batch_size 1 --learning_rate 3e-3 --warmup_steps 500 \
> --predict_with_generate --save_steps 0 --eval_steps 0 --group_by_length \
> --dataset_name wmt16 --dataset_config ro-en --source_lang en --target_lang ro \
> --source_prefix 'translate English to Romanian: ' --logging_steps 5 --logging_strategy \
> steps
> ```

Thank you for your reply. Our current machine is undergoing model training. This is expected to take 3 days. As I currently do not have any additional machines to test, I will try them in 3 days.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey @MikeDean2367, just wanted to know if and how you solved the tqdm/progress bar issue. I am getting the same issue under similar setup where the progress bar shows up after the model has finished training.

Seeing similar behavior running Deepspeed with accelerate to fine tune a llama2-70b model.

I tested the DeepSpeed startup in both multi-node and single-node multi-GPU scenarios and found that the former did not display a progress bar.

For the display of the progress bar, I encountered the same issue, and I found that a simple fix is that modify "deepspeed --num_gpus 8 --num_nodes 2  --hostfile=host.txt train.py" into "torchrun --nnodes 2 --nproc_per_node 8 --master_addr xxx.xxx.xxx.xxx".
In other word, launch with deepspeed mode would dismiss the progress bar while torchrun mode not.

Has anyone solved the problem about deepspeed launch trainer but progressing bar do not show?

> For the display of the progress bar, I encountered the same issue, and I found that a simple fix is that modify "deepspeed --num_gpus 8 --num_nodes 2 --hostfile=host.txt train.py" into "torchrun --nnodes 2 --nproc_per_node 8 --master_addr xxx.xxx.xxx.xxx". In other word, launch with deepspeed mode would dismiss the progress bar while torchrun mode not.

I encountered the same issue, and when using pdsh to start torchrun in a distributed manner, the progress bar does not appear either.

same ! no progress bar, or rather, it will not get updated

Same problem when using pdsh.

Temp workaround is to print or log the progress bar manually, instead of relying on the .write() method because the bar doesn't render through a SSH session.