Pre-build DeepSpeed
Author: ydshieh
Pull request: yes

# What does this PR do?

Pre-build `DeepSpeed`, so the first deepspeed test won't timeout.

In the recent change, I installed it as `pip install deepspeed`, and we got a CI failure
```
# This one timed out
tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_can_resume_training_normal_zero2_fp16
```

See [this comment](https://github.com/huggingface/transformers/pull/17417#issuecomment-1148907094) for more details.

Comments:
@stas00 

I changed what you provided
```
DS_BUILD_CPU_ADAM=1 DS_BUILD_AIO=1 DS_BUILD_UTILS=1 pip install -e . \
--global-option="build_ext" --global-option="-j8"
```
to
```
DS_BUILD_CPU_ADAM=1 DS_BUILD_AIO=1 DS_BUILD_UTILS=1 python3 -m pip install deepspeed \
--global-option="build_ext" --global-option="-j8" --no-cache -v --disable-pip-version-check
```
as we want to use the latest release, instead of the nightly `DeepSpeed`.
(other flags are just copied from the previous version of our docker file)

Verified this command inside docker, and it compiles successfully.
Ran the tests, and it looks very speedy!

_The documentation is not available anymore as the PR was closed or merged._

(FMI)

Documentation about DeepSpeed Pre-Install
https://www.deepspeed.ai/tutorials/advanced-install/#pre-install-deepspeed-ops

`python3 -m ` is probably redundant, but otherwise looks good.

oh, don't you need `-U` there as well? as if it's already installed it'll get stuck at that version.

> `python3 -m ` is probably redundant, but otherwise looks good.
> 
> oh, don't you need `-U` there as well? as if it's already installed it'll get stuck at that version.

I always forgot `-U` and used `python3 -m pip uninstall -y deepspeed` before installing it again.
I will change it to `-U`

while uninstall works, `-U` makes things more atomic and easier to not make a mistake of forgetting to uninstall especially when your build file has many entries. But either way works fine.

I tried with `-U` and it won't build DeepSpeed: 
```
Requirement already satisfied: deepspeed in /opt/conda/lib/python3.8/site-packages (0.6.5)
```
So if we want to **pre-build** `DeepSpeed`, it's better to uninstall first.


but isn't it because you already had it prebuilt and there is no newer version? so of course it skips the building since it was already built.

Unless you are using my recipe of first installing `pip install deepspeed` to get the dependencies in (it breaks with prebuilding as it tries to build each heavy binary dependency from source). In which case yes, the recipe would be:

```
# arrange for deepspeed deps install
pip install deepspeed 
# now build from source, which requires removing it first
pip uninstall deepspeed -y
DS_BUILD_CPU_ADAM=1 DS_BUILD_AIO=1 DS_BUILD_UTILS=1 pip install deepspeed ...
```
it's too bad `pip` doesn't have a flag to just install the deps of a given package and not the package.


So far (current main), it is not pre-built.

```
# I just kept this as before. It also installs everything in `extras["testing"] + extras["optuna"]`
RUN python3 -m pip install --no-cache-dir -e ./transformers[deepspeed-testing]

RUN python3 -m pip uninstall -y deepspeed
RUN DS_BUILD_CPU_ADAM=1 DS_BUILD_AIO=1 DS_BUILD_UTILS=1 python3 -m pip install deepspeed --global-option="build_ext" --global-option="-j8" --no-cache -v --disable-pip-version-check
```

So yes, it is using your recipe (at least in a similar sense).

what's the point of `-e` here? This should be sufficient:

```
RUN python3 -m pip install --no-cache-dir  transformers[deepspeed-testing]
```

the rest looks good.

It's quite strange that pre-build deepspeed during docker image build will have some tests failed.

If I do pre-building when the docker running, but before the tests run, it is working.

I remember in the previous version (before all my  work on docker and workflow), Lysandre was doing the same thing.
I will make this change though.

I will also post the error message to keep the info recorded.

is is possible that the docker is run with different options? i.e. when it works and when it doesn't

The docker runs have the same options - even the exact command to launch:
```bash
docker run --gpus all --shm-size "16gb" --ipc host -it huggingface/transformers-pytorch-deepspeed-latest-gpu-test
```

The very long full output of the test below is with the command
```bash
TRANSFORMERS_IS_CI=yes OMP_NUM_THREADS=8 MKL_NUM_THREADS=8 RUN_SLOW=yes TF_FORCE_GPU_ALLOW_GROWTH=true RUN_PT_TF_CROSS_TESTS=1 python3 -m pytest -sv --make-reports=tests_gpu  tests/deepspeed -k "TestDeepSpeedWithLauncher and test_basic_distributed_zero2_fp16" --durations=0
```
(I only tested with `test_basic_distributed_zero2_fp16`)
(I also randomly tested another one: `test_init_zero3_fp16`, which is fine)

As you can see in the last commit, I just **pre-build DeepSpeed again in the workflow file**, and [it works fine](https://github.com/huggingface/transformers/actions/runs/2470871622)
**I will merge this PR now**, but keep the output below. 

It would be super if you have immediate insights from this output. Otherwise I think it's fine to focus on other priorities for now.
Let me know if I can get more detailed outputs for you to look. (I haven't tried `py-spy dump --pid PID` yet) 

## **Error without (uninstall first) pre-building again inside docker running**
(I saw `stdout: [2022-06-09 20:27:17,594] [INFO] [launch.py:178:sigkill_handler] Killing subprocess 489`)
```
========================================================================================= test session starts ==========================================================================================
platform linux -- Python 3.8.8, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /opt/conda/bin/python3
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/workspace/transformers/.hypothesis/examples')
rootdir: /workspace/transformers, configfile: setup.cfg
plugins: xdist-2.5.0, forked-1.4.0, timeout-2.1.0, cov-2.11.1, pythonpath-0.7.3, hypothesis-4.50.8
collected 140 items / 139 deselected / 1 selected                                                                                                                                                      

tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_basic_distributed_zero2_fp16 
Running:  deepspeed --num_nodes 1 --num_gpus 1 --master_port 10999 /workspace/transformers/examples/pytorch/translation/run_translation.py --model_name_or_path t5-small --train_file /workspace/transformers/tests/deepspeed/../fixtures/tests_samples/wmt_en_ro/train.json --validation_file /workspace/transformers/tests/deepspeed/../fixtures/tests_samples/wmt_en_ro/val.json --output_dir /tmp/tmpboxxylsc --overwrite_output_dir --max_source_length 32 --max_target_length 32 --val_max_target_length 32 --warmup_steps 8 --predict_with_generate --save_steps 0 --eval_steps 10 --group_by_length --label_smoothing_factor 0.1 --source_lang en --target_lang ro --report_to none --source_prefix "translate English to Romanian: " --fp16 --do_train --num_train_epochs 1 --max_train_samples 16 --per_device_train_batch_size 2 --learning_rate 3e-3 --do_eval --max_eval_samples 16 --per_device_eval_batch_size 2 --deepspeed /workspace/transformers/tests/deepspeed/ds_config_zero2.json
stdout: [2022-06-09 20:27:03,709] [WARNING] [runner.py:159:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
stdout: [2022-06-09 20:27:03,717] [INFO] [runner.py:457:main] cmd = /opt/conda/bin/python3 -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMF19 --master_addr=127.0.0.1 --master_port=10999 /workspace/transformers/examples/pytorch/translation/run_translation.py --model_name_or_path t5-small --train_file /workspace/transformers/tests/deepspeed/../fixtures/tests_samples/wmt_en_ro/train.json --validation_file /workspace/transformers/tests/deepspeed/../fixtures/tests_samples/wmt_en_ro/val.json --output_dir /tmp/tmpboxxylsc --overwrite_output_dir --max_source_length 32 --max_target_length 32 --val_max_target_length 32 --warmup_steps 8 --predict_with_generate --save_steps 0 --eval_steps 10 --group_by_length --label_smoothing_factor 0.1 --source_lang en --target_lang ro --report_to none --source_prefix "translate English to Romanian: " --fp16 --do_train --num_train_epochs 1 --max_train_samples 16 --per_device_train_batch_size 2 --learning_rate 3e-3 --do_eval --max_eval_samples 16 --per_device_eval_batch_size 2 --deepspeed /workspace/transformers/tests/deepspeed/ds_config_zero2.json
stdout: [2022-06-09 20:27:04,574] [INFO] [launch.py:96:main] 0 NCCL_VERSION=2.8.4
stdout: [2022-06-09 20:27:04,574] [INFO] [launch.py:103:main] WORLD INFO DICT: {'localhost': [0]}
stdout: [2022-06-09 20:27:04,574] [INFO] [launch.py:109:main] nnodes=1, num_local_procs=1, node_rank=0
stdout: [2022-06-09 20:27:04,574] [INFO] [launch.py:122:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0]})
stdout: [2022-06-09 20:27:04,574] [INFO] [launch.py:123:main] dist_world_size=1
stdout: [2022-06-09 20:27:04,574] [INFO] [launch.py:125:main] Setting CUDA_VISIBLE_DEVICES=0
stdout: [2022-06-09 20:27:06,867] [INFO] [distributed.py:48:init_distributed] Initializing torch distributed with backend: nccl
stdout: 06/09/2022 20:27:07 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: True
stdout: 06/09/2022 20:27:07 - INFO - __main__ - Training/evaluation parameters Seq2SeqTrainingArguments(
stdout: _n_gpu=1,
stdout: adafactor=False,
stdout: adam_beta1=0.9,
stdout: adam_beta2=0.999,
stdout: adam_epsilon=1e-08,
stdout: auto_find_batch_size=False,
stdout: bf16=False,
stdout: bf16_full_eval=False,
stdout: data_seed=None,
stdout: dataloader_drop_last=False,
stdout: dataloader_num_workers=0,
stdout: dataloader_pin_memory=True,
stdout: ddp_bucket_cap_mb=None,
stdout: ddp_find_unused_parameters=None,
stdout: debug=[],
stdout: deepspeed=/workspace/transformers/tests/deepspeed/ds_config_zero2.json,
stdout: disable_tqdm=False,
stdout: do_eval=True,
stdout: do_predict=False,
stdout: do_train=True,
stdout: eval_accumulation_steps=None,
stdout: eval_delay=0,
stdout: eval_steps=10,
stdout: evaluation_strategy=IntervalStrategy.NO,
stdout: fp16=True,
stdout: fp16_backend=auto,
stdout: fp16_full_eval=False,
stdout: fp16_opt_level=O1,
stdout: fsdp=[],
stdout: fsdp_min_num_params=0,
stdout: full_determinism=False,
stdout: generation_max_length=None,
stdout: generation_num_beams=None,
stdout: gradient_accumulation_steps=1,
stdout: gradient_checkpointing=False,
stdout: greater_is_better=None,
stdout: group_by_length=True,
stdout: half_precision_backend=auto,
stdout: hub_model_id=None,
stdout: hub_private_repo=False,
stdout: hub_strategy=HubStrategy.EVERY_SAVE,
stdout: hub_token=<HUB_TOKEN>,
stdout: ignore_data_skip=False,
stdout: include_inputs_for_metrics=False,
stdout: label_names=None,
stdout: label_smoothing_factor=0.1,
stdout: learning_rate=0.003,
stdout: length_column_name=length,
stdout: load_best_model_at_end=False,
stdout: local_rank=0,
stdout: log_level=-1,
stdout: log_level_replica=-1,
stdout: log_on_each_node=True,
stdout: logging_dir=/tmp/tmpboxxylsc/runs/Jun09_20-27-06_1e3412729191,
stdout: logging_first_step=False,
stdout: logging_nan_inf_filter=True,
stdout: logging_steps=500,
stdout: logging_strategy=IntervalStrategy.STEPS,
stdout: lr_scheduler_type=SchedulerType.LINEAR,
stdout: max_grad_norm=1.0,
stdout: max_steps=-1,
stdout: metric_for_best_model=None,
stdout: mp_parameters=,
stdout: no_cuda=False,
stdout: num_train_epochs=1.0,
stdout: optim=OptimizerNames.ADAMW_HF,
stdout: output_dir=/tmp/tmpboxxylsc,
stdout: overwrite_output_dir=True,
stdout: past_index=-1,
stdout: per_device_eval_batch_size=2,
stdout: per_device_train_batch_size=2,
stdout: predict_with_generate=True,
stdout: prediction_loss_only=False,
stdout: push_to_hub=False,
stdout: push_to_hub_model_id=None,
stdout: push_to_hub_organization=None,
stdout: push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
stdout: remove_unused_columns=True,
stdout: report_to=[],
stdout: resume_from_checkpoint=None,
stdout: run_name=/tmp/tmpboxxylsc,
stdout: save_on_each_node=False,
stdout: save_steps=0,
stdout: save_strategy=IntervalStrategy.STEPS,
stdout: save_total_limit=None,
stdout: seed=42,
stdout: sharded_ddp=[],
stdout: skip_memory_metrics=True,
stdout: sortish_sampler=False,
stdout: tf32=None,
stdout: torchdynamo=None,
stdout: tpu_metrics_debug=False,
stdout: tpu_num_cores=None,
stdout: use_ipex=False,
stdout: use_legacy_prediction_loop=False,
stdout: warmup_ratio=0.0,
stdout: warmup_steps=8,
stdout: weight_decay=0.0,
stdout: xpu_backend=None,
stdout: )
stdout: 06/09/2022 20:27:07 - WARNING - datasets.builder - Using custom data configuration default-fad90dc19745c6b9
stdout: 06/09/2022 20:27:07 - INFO - datasets.builder - Overwrite dataset info from restored data version.
stdout: 06/09/2022 20:27:07 - INFO - datasets.info - Loading Dataset info from /root/.cache/huggingface/datasets/json/default-fad90dc19745c6b9/0.0.0/ac0ca5f5289a6cf108e706efcf040422dbbfa8e658dee6a819f20d76bb84d26b
stdout: 06/09/2022 20:27:07 - WARNING - datasets.builder - Reusing dataset json (/root/.cache/huggingface/datasets/json/default-fad90dc19745c6b9/0.0.0/ac0ca5f5289a6cf108e706efcf040422dbbfa8e658dee6a819f20d76bb84d26b)
stdout: 06/09/2022 20:27:07 - INFO - datasets.info - Loading Dataset info from /root/.cache/huggingface/datasets/json/default-fad90dc19745c6b9/0.0.0/ac0ca5f5289a6cf108e706efcf040422dbbfa8e658dee6a819f20d76bb84d26b
100%|██████████| 2/2 [00:00<00:00, 752.81it/s]
stderr: [INFO|configuration_utils.py:659] 2022-06-09 20:27:07,430 >> loading configuration file https://huggingface.co/t5-small/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/fe501e8fd6425b8ec93df37767fcce78ce626e34cc5edc859c662350cf712e41.406701565c0afd9899544c1cb8b93185a76f00b31e5ce7f6e18bbaef02241985
stderr: [INFO|configuration_utils.py:708] 2022-06-09 20:27:07,444 >> Model config T5Config {
stderr:   "_name_or_path": "t5-small",
stderr:   "architectures": [
stderr:     "T5WithLMHeadModel"
stderr:   ],
stderr:   "d_ff": 2048,
stderr:   "d_kv": 64,
stderr:   "d_model": 512,
stderr:   "decoder_start_token_id": 0,
stderr:   "dense_act_fn": "relu",
stderr:   "dropout_rate": 0.1,
stderr:   "eos_token_id": 1,
stderr:   "feed_forward_proj": "relu",
stderr:   "initializer_factor": 1.0,
stderr:   "is_encoder_decoder": true,
stderr:   "is_gated_act": false,
stderr:   "layer_norm_epsilon": 1e-06,
stderr:   "model_type": "t5",
stderr:   "n_positions": 512,
stderr:   "num_decoder_layers": 6,
stderr:   "num_heads": 8,
stderr:   "num_layers": 6,
stderr:   "output_past": true,
stderr:   "pad_token_id": 0,
stderr:   "relative_attention_max_distance": 128,
stderr:   "relative_attention_num_buckets": 32,
stderr:   "task_specific_params": {
stderr:     "summarization": {
stderr:       "early_stopping": true,
stderr:       "length_penalty": 2.0,
stderr:       "max_length": 200,
stderr:       "min_length": 30,
stderr:       "no_repeat_ngram_size": 3,
stderr:       "num_beams": 4,
stderr:       "prefix": "summarize: "
stderr:     },
stderr:     "translation_en_to_de": {
stderr:       "early_stopping": true,
stderr:       "max_length": 300,
stderr:       "num_beams": 4,
stderr:       "prefix": "translate English to German: "
stderr:     },
stderr:     "translation_en_to_fr": {
stderr:       "early_stopping": true,
stderr:       "max_length": 300,
stderr:       "num_beams": 4,
stderr:       "prefix": "translate English to French: "
stderr:     },
stderr:     "translation_en_to_ro": {
stderr:       "early_stopping": true,
stderr:       "max_length": 300,
stderr:       "num_beams": 4,
stderr:       "prefix": "translate English to Romanian: "
stderr:     }
stderr:   },
stderr:   "transformers_version": "4.20.0.dev0",
stderr:   "use_cache": true,
stderr:   "vocab_size": 32128
stderr: }
stderr: 
stderr: [INFO|tokenization_auto.py:384] 2022-06-09 20:27:07,585 >> Could not locate the tokenizer configuration file, will try to use the model config instead.
stderr: [INFO|configuration_utils.py:659] 2022-06-09 20:27:07,723 >> loading configuration file https://huggingface.co/t5-small/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/fe501e8fd6425b8ec93df37767fcce78ce626e34cc5edc859c662350cf712e41.406701565c0afd9899544c1cb8b93185a76f00b31e5ce7f6e18bbaef02241985
stderr: [INFO|configuration_utils.py:708] 2022-06-09 20:27:07,724 >> Model config T5Config {
stderr:   "_name_or_path": "t5-small",
stderr:   "architectures": [
stderr:     "T5WithLMHeadModel"
stderr:   ],
stderr:   "d_ff": 2048,
stderr:   "d_kv": 64,
stderr:   "d_model": 512,
stderr:   "decoder_start_token_id": 0,
stderr:   "dense_act_fn": "relu",
stderr:   "dropout_rate": 0.1,
stderr:   "eos_token_id": 1,
stderr:   "feed_forward_proj": "relu",
stderr:   "initializer_factor": 1.0,
stderr:   "is_encoder_decoder": true,
stderr:   "is_gated_act": false,
stderr:   "layer_norm_epsilon": 1e-06,
stderr:   "model_type": "t5",
stderr:   "n_positions": 512,
stderr:   "num_decoder_layers": 6,
stderr:   "num_heads": 8,
stderr:   "num_layers": 6,
stderr:   "output_past": true,
stderr:   "pad_token_id": 0,
stderr:   "relative_attention_max_distance": 128,
stderr:   "relative_attention_num_buckets": 32,
stderr:   "task_specific_params": {
stderr:     "summarization": {
stderr:       "early_stopping": true,
stderr:       "length_penalty": 2.0,
stderr:       "max_length": 200,
stderr:       "min_length": 30,
stderr:       "no_repeat_ngram_size": 3,
stderr:       "num_beams": 4,
stderr:       "prefix": "summarize: "
stderr:     },
stderr:     "translation_en_to_de": {
stderr:       "early_stopping": true,
stderr:       "max_length": 300,
stderr:       "num_beams": 4,
stderr:       "prefix": "translate English to German: "
stderr:     },
stderr:     "translation_en_to_fr": {
stderr:       "early_stopping": true,
stderr:       "max_length": 300,
stderr:       "num_beams": 4,
stderr:       "prefix": "translate English to French: "
stderr:     },
stderr:     "translation_en_to_ro": {
stderr:       "early_stopping": true,
stderr:       "max_length": 300,
stderr:       "num_beams": 4,
stderr:       "prefix": "translate English to Romanian: "
stderr:     }
stderr:   },
stderr:   "transformers_version": "4.20.0.dev0",
stderr:   "use_cache": true,
stderr:   "vocab_size": 32128
stderr: }
stderr: 
stderr: [INFO|tokenization_utils_base.py:1781] 2022-06-09 20:27:08,559 >> loading file https://huggingface.co/t5-small/resolve/main/spiece.model from cache at /root/.cache/huggingface/transformers/65fc04e21f45f61430aea0c4fedffac16a4d20d78b8e6601d8d996ebefefecd2.3b69006860e7b5d0a63ffdddc01ddcd6b7c318a6f4fd793596552c741734c62d
stderr: [INFO|tokenization_utils_base.py:1781] 2022-06-09 20:27:08,559 >> loading file https://huggingface.co/t5-small/resolve/main/tokenizer.json from cache at /root/.cache/huggingface/transformers/06779097c78e12f47ef67ecb728810c2ae757ee0a9efe9390c6419783d99382d.8627f1bd5d270a9fd2e5a51c8bec3223896587cc3cfe13edeabb0992ab43c529
stderr: [INFO|tokenization_utils_base.py:1781] 2022-06-09 20:27:08,559 >> loading file https://huggingface.co/t5-small/resolve/main/added_tokens.json from cache at None
stderr: [INFO|tokenization_utils_base.py:1781] 2022-06-09 20:27:08,559 >> loading file https://huggingface.co/t5-small/resolve/main/special_tokens_map.json from cache at None
stderr: [INFO|tokenization_utils_base.py:1781] 2022-06-09 20:27:08,559 >> loading file https://huggingface.co/t5-small/resolve/main/tokenizer_config.json from cache at None
stderr: [INFO|configuration_utils.py:659] 2022-06-09 20:27:08,700 >> loading configuration file https://huggingface.co/t5-small/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/fe501e8fd6425b8ec93df37767fcce78ce626e34cc5edc859c662350cf712e41.406701565c0afd9899544c1cb8b93185a76f00b31e5ce7f6e18bbaef02241985
stderr: [INFO|configuration_utils.py:708] 2022-06-09 20:27:08,701 >> Model config T5Config {
stderr:   "_name_or_path": "t5-small",
stderr:   "architectures": [
stderr:     "T5WithLMHeadModel"
stderr:   ],
stderr:   "d_ff": 2048,
stderr:   "d_kv": 64,
stderr:   "d_model": 512,
stderr:   "decoder_start_token_id": 0,
stderr:   "dense_act_fn": "relu",
stderr:   "dropout_rate": 0.1,
stderr:   "eos_token_id": 1,
stderr:   "feed_forward_proj": "relu",
stderr:   "initializer_factor": 1.0,
stderr:   "is_encoder_decoder": true,
stderr:   "is_gated_act": false,
stderr:   "layer_norm_epsilon": 1e-06,
stderr:   "model_type": "t5",
stderr:   "n_positions": 512,
stderr:   "num_decoder_layers": 6,
stderr:   "num_heads": 8,
stderr:   "num_layers": 6,
stderr:   "output_past": true,
stderr:   "pad_token_id": 0,
stderr:   "relative_attention_max_distance": 128,
stderr:   "relative_attention_num_buckets": 32,
stderr:   "task_specific_params": {
stderr:     "summarization": {
stderr:       "early_stopping": true,
stderr:       "length_penalty": 2.0,
stderr:       "max_length": 200,
stderr:       "min_length": 30,
stderr:       "no_repeat_ngram_size": 3,
stderr:       "num_beams": 4,
stderr:       "prefix": "summarize: "
stderr:     },
stderr:     "translation_en_to_de": {
stderr:       "early_stopping": true,
stderr:       "max_length": 300,
stderr:       "num_beams": 4,
stderr:       "prefix": "translate English to German: "
stderr:     },
stderr:     "translation_en_to_fr": {
stderr:       "early_stopping": true,
stderr:       "max_length": 300,
stderr:       "num_beams": 4,
stderr:       "prefix": "translate English to French: "
stderr:     },
stderr:     "translation_en_to_ro": {
stderr:       "early_stopping": true,
stderr:       "max_length": 300,
stderr:       "num_beams": 4,
stderr:       "prefix": "translate English to Romanian: "
stderr:     }
stderr:   },
stderr:   "transformers_version": "4.20.0.dev0",
stderr:   "use_cache": true,
stderr:   "vocab_size": 32128
stderr: }
stderr: 
stderr: /workspace/transformers/src/transformers/models/t5/tokenization_t5_fast.py:156: FutureWarning: This tokenizer was incorrectly instantiated with a model max length of 512 which will be corrected in Transformers v5.
stderr: For now, this behavior is kept to avoid breaking backwards compatibility when padding/encoding with `truncation is True`.
stderr: - Be aware that you SHOULD NOT rely on t5-small automatically truncating your input to 512 when padding/encoding.
stderr: - If you want to encode/pad to sequences longer than 512 you can either instantiate this tokenizer with `model_max_length` or pass `max_length` when encoding/padding.
stderr: - To avoid this warning, please instantiate this tokenizer with `model_max_length` set to your preferred value.
stderr:   warnings.warn(
stderr: [INFO|modeling_utils.py:2049] 2022-06-09 20:27:08,937 >> loading weights file https://huggingface.co/t5-small/resolve/main/pytorch_model.bin from cache at /root/.cache/huggingface/transformers/fee5a3a0ae379232608b6eed45d2d7a0d2966b9683728838412caccc41b4b0ed.ddacdc89ec88482db20c676f0861a336f3d0409f94748c209847b49529d73885
stderr: [INFO|modeling_utils.py:2425] 2022-06-09 20:27:09,933 >> All model checkpoint weights were used when initializing T5ForConditionalGeneration.
stderr: 
stderr: [INFO|modeling_utils.py:2433] 2022-06-09 20:27:09,933 >> All the weights of T5ForConditionalGeneration were initialized from the model checkpoint at t5-small.
stderr: If your task is similar to the task the model of the checkpoint was trained on, you can already use T5ForConditionalGeneration for predictions without further training.
stdout: 06/09/2022 20:27:10 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /root/.cache/huggingface/datasets/json/default-fad90dc19745c6b9/0.0.0/ac0ca5f5289a6cf108e706efcf040422dbbfa8e658dee6a819f20d76bb84d26b/cache-94ae3814baee36bf.arrow
stdout: 06/09/2022 20:27:10 - INFO - datasets.arrow_dataset - Caching processed dataset at /root/.cache/huggingface/datasets/json/default-fad90dc19745c6b9/0.0.0/ac0ca5f5289a6cf108e706efcf040422dbbfa8e658dee6a819f20d76bb84d26b/cache-f0e9ed0becfce7c3.arrow
Running tokenizer on validation dataset: 100%|██████████| 1/1 [00:00<00:00, 60.72ba/s]
stderr: [INFO|trainer.py:519] 2022-06-09 20:27:10,499 >> Using cuda_amp half precision backend
stdout: [2022-06-09 20:27:10,506] [INFO] [logging.py:69:log_dist] [Rank 0] DeepSpeed info: version=0.6.5, git-hash=unknown, git-branch=unknown
stdout: [2022-06-09 20:27:14,861] [INFO] [engine.py:278:__init__] DeepSpeed Flops Profiler Enabled: False
stdout: Adam Optimizer #0 is created with AVX2 arithmetic capability.
stdout: Config: alpha=0.003000, betas=(0.900000, 0.999000), weight_decay=0.000000, adam_w=1
stdout: [2022-06-09 20:27:15,316] [INFO] [engine.py:1100:_configure_optimizer] Using DeepSpeed Optimizer param name adamw as basic optimizer
stdout: [2022-06-09 20:27:15,321] [INFO] [engine.py:1108:_configure_optimizer] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam
stdout: [2022-06-09 20:27:15,321] [INFO] [utils.py:52:is_zero_supported_optimizer] Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'>
stdout: [2022-06-09 20:27:15,321] [INFO] [logging.py:69:log_dist] [Rank 0] Creating fp16 ZeRO stage 2 optimizer
stdout: [2022-06-09 20:27:15,321] [INFO] [stage_1_and_2.py:133:__init__] Reduce bucket size 200000000.0
stdout: [2022-06-09 20:27:15,321] [INFO] [stage_1_and_2.py:134:__init__] Allgather bucket size 200000000.0
stdout: [2022-06-09 20:27:15,321] [INFO] [stage_1_and_2.py:135:__init__] CPU Offload: True
stdout: [2022-06-09 20:27:15,321] [INFO] [stage_1_and_2.py:136:__init__] Round robin gradient partitioning: False
stdout: Rank: 0 partition count [1] and sizes[(60492288, False)]
stdout: [2022-06-09 20:27:15,709] [INFO] [utils.py:828:see_memory_usage] Before initializing optimizer states
stdout: [2022-06-09 20:27:15,710] [INFO] [utils.py:829:see_memory_usage] MA 0.14 GB         Max_MA 0.14 GB         CA 0.24 GB         Max_CA 0 GB
stdout: [2022-06-09 20:27:15,710] [INFO] [utils.py:837:see_memory_usage] CPU Virtual Memory:  used = 6.3 GB, percent = 10.7%
stdout: [2022-06-09 20:27:17,594] [INFO] [launch.py:178:sigkill_handler] Killing subprocess 489
stdout: [2022-06-09 20:27:17,594] [ERROR] [launch.py:184:sigkill_handler] ['/opt/conda/bin/python3', '-u', '/workspace/transformers/examples/pytorch/translation/run_translation.py', '--local_rank=0', '--model_name_or_path', 't5-small', '--train_file', '/workspace/transformers/tests/deepspeed/../fixtures/tests_samples/wmt_en_ro/train.json', '--validation_file', '/workspace/transformers/tests/deepspeed/../fixtures/tests_samples/wmt_en_ro/val.json', '--output_dir', '/tmp/tmpboxxylsc', '--overwrite_output_dir', '--max_source_length', '32', '--max_target_length', '32', '--val_max_target_length', '32', '--warmup_steps', '8', '--predict_with_generate', '--save_steps', '0', '--eval_steps', '10', '--group_by_length', '--label_smoothing_factor', '0.1', '--source_lang', 'en', '--target_lang', 'ro', '--report_to', 'none', '--source_prefix', '"translate English to Romanian: "', '--fp16', '--do_train', '--num_train_epochs', '1', '--max_train_samples', '16', '--per_device_train_batch_size', '2', '--learning_rate', '3e-3', '--do_eval', '--max_eval_samples', '16', '--per_device_eval_batch_size', '2', '--deepspeed', '/workspace/transformers/tests/deepspeed/ds_config_zero2.json'] exits with return code = -4
FAILED

=============================================================================================== FAILURES ===============================================================================================
_____________________________________________________________________ TestDeepSpeedWithLauncher.test_basic_distributed_zero2_fp16 ______________________________________________________________________

a = (<test_deepspeed.TestDeepSpeedWithLauncher testMethod=test_basic_distributed_zero2_fp16>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

/opt/conda/lib/python3.8/site-packages/parameterized/parameterized.py:533: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/deepspeed/test_deepspeed.py:858: in test_basic_distributed
    self.run_and_check(stage=stage, dtype=dtype, distributed=True)
tests/deepspeed/test_deepspeed.py:971: in run_and_check
    output_dir = self.run_trainer(
tests/deepspeed/test_deepspeed.py:1070: in run_trainer
    execute_subprocess_async(cmd, env=self.get_env())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cmd = ['deepspeed', '--num_nodes', '1', '--num_gpus', '1', '--master_port', ...]
env = {'BASH_ENV': '/etc/bash.bashrc', 'COCOAPI_VERSION': '2.0+nv0.4.0', 'CUBLAS_VERSION': '11.4.1.1026', 'CUDA_CACHE_DISABLE': '1', ...}, stdin = None, timeout = 180, quiet = False, echo = True

    def execute_subprocess_async(cmd, env=None, stdin=None, timeout=180, quiet=False, echo=True) -> _RunOutput:
    
        loop = asyncio.get_event_loop()
        result = loop.run_until_complete(
            _stream_subprocess(cmd, env=env, stdin=stdin, timeout=timeout, quiet=quiet, echo=echo)
        )
    
        cmd_str = " ".join(cmd)
        if result.returncode > 0:
            stderr = "\n".join(result.stderr)
>           raise RuntimeError(
                f"'{cmd_str}' failed with returncode {result.returncode}\n\n"
                f"The combined stderr from workers follows:\n{stderr}"
            )
E           RuntimeError: 'deepspeed --num_nodes 1 --num_gpus 1 --master_port 10999 /workspace/transformers/examples/pytorch/translation/run_translation.py --model_name_or_path t5-small --train_file /workspace/transformers/tests/deepspeed/../fixtures/tests_samples/wmt_en_ro/train.json --validation_file /workspace/transformers/tests/deepspeed/../fixtures/tests_samples/wmt_en_ro/val.json --output_dir /tmp/tmpboxxylsc --overwrite_output_dir --max_source_length 32 --max_target_length 32 --val_max_target_length 32 --warmup_steps 8 --predict_with_generate --save_steps 0 --eval_steps 10 --group_by_length --label_smoothing_factor 0.1 --source_lang en --target_lang ro --report_to none --source_prefix "translate English to Romanian: " --fp16 --do_train --num_train_epochs 1 --max_train_samples 16 --per_device_train_batch_size 2 --learning_rate 3e-3 --do_eval --max_eval_samples 16 --per_device_eval_batch_size 2 --deepspeed /workspace/transformers/tests/deepspeed/ds_config_zero2.json' failed with returncode 252
E           
E           The combined stderr from workers follows:
100%|██████████| 2/2 [00:00<00:00, 752.81it/s]
E           [INFO|configuration_utils.py:659] 2022-06-09 20:27:07,430 >> loading configuration file https://huggingface.co/t5-small/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/fe501e8fd6425b8ec93df37767fcce78ce626e34cc5edc859c662350cf712e41.406701565c0afd9899544c1cb8b93185a76f00b31e5ce7f6e18bbaef02241985
E           [INFO|configuration_utils.py:708] 2022-06-09 20:27:07,444 >> Model config T5Config {
E             "_name_or_path": "t5-small",
E             "architectures": [
E               "T5WithLMHeadModel"
E             ],
E             "d_ff": 2048,
E             "d_kv": 64,
E             "d_model": 512,
E             "decoder_start_token_id": 0,
E             "dense_act_fn": "relu",
E             "dropout_rate": 0.1,
E             "eos_token_id": 1,
E             "feed_forward_proj": "relu",
E             "initializer_factor": 1.0,
E             "is_encoder_decoder": true,
E             "is_gated_act": false,
E             "layer_norm_epsilon": 1e-06,
E             "model_type": "t5",
E             "n_positions": 512,
E             "num_decoder_layers": 6,
E             "num_heads": 8,
E             "num_layers": 6,
E             "output_past": true,
E             "pad_token_id": 0,
E             "relative_attention_max_distance": 128,
E             "relative_attention_num_buckets": 32,
E             "task_specific_params": {
E               "summarization": {
E                 "early_stopping": true,
E                 "length_penalty": 2.0,
E                 "max_length": 200,
E                 "min_length": 30,
E                 "no_repeat_ngram_size": 3,
E                 "num_beams": 4,
E                 "prefix": "summarize: "
E               },
E               "translation_en_to_de": {
E                 "early_stopping": true,
E                 "max_length": 300,
E                 "num_beams": 4,
E                 "prefix": "translate English to German: "
E               },
E               "translation_en_to_fr": {
E                 "early_stopping": true,
E                 "max_length": 300,
E                 "num_beams": 4,
E                 "prefix": "translate English to French: "
E               },
E               "translation_en_to_ro": {
E                 "early_stopping": true,
E                 "max_length": 300,
E                 "num_beams": 4,
E                 "prefix": "translate English to Romanian: "
E               }
E             },
E             "transformers_version": "4.20.0.dev0",
E             "use_cache": true,
E             "vocab_size": 32128
E           }
E           
E           [INFO|tokenization_auto.py:384] 2022-06-09 20:27:07,585 >> Could not locate the tokenizer configuration file, will try to use the model config instead.
E           [INFO|configuration_utils.py:659] 2022-06-09 20:27:07,723 >> loading configuration file https://huggingface.co/t5-small/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/fe501e8fd6425b8ec93df37767fcce78ce626e34cc5edc859c662350cf712e41.406701565c0afd9899544c1cb8b93185a76f00b31e5ce7f6e18bbaef02241985
E           [INFO|configuration_utils.py:708] 2022-06-09 20:27:07,724 >> Model config T5Config {
E             "_name_or_path": "t5-small",
E             "architectures": [
E               "T5WithLMHeadModel"
E             ],
E             "d_ff": 2048,
E             "d_kv": 64,
E             "d_model": 512,
E             "decoder_start_token_id": 0,
E             "dense_act_fn": "relu",
E             "dropout_rate": 0.1,
E             "eos_token_id": 1,
E             "feed_forward_proj": "relu",
E             "initializer_factor": 1.0,
E             "is_encoder_decoder": true,
E             "is_gated_act": false,
E             "layer_norm_epsilon": 1e-06,
E             "model_type": "t5",
E             "n_positions": 512,
E             "num_decoder_layers": 6,
E             "num_heads": 8,
E             "num_layers": 6,
E             "output_past": true,
E             "pad_token_id": 0,
E             "relative_attention_max_distance": 128,
E             "relative_attention_num_buckets": 32,
E             "task_specific_params": {
E               "summarization": {
E                 "early_stopping": true,
E                 "length_penalty": 2.0,
E                 "max_length": 200,
E                 "min_length": 30,
E                 "no_repeat_ngram_size": 3,
E                 "num_beams": 4,
E                 "prefix": "summarize: "
E               },
E               "translation_en_to_de": {
E                 "early_stopping": true,
E                 "max_length": 300,
E                 "num_beams": 4,
E                 "prefix": "translate English to German: "
E               },
E               "translation_en_to_fr": {
E                 "early_stopping": true,
E                 "max_length": 300,
E                 "num_beams": 4,
E                 "prefix": "translate English to French: "
E               },
E               "translation_en_to_ro": {
E                 "early_stopping": true,
E                 "max_length": 300,
E                 "num_beams": 4,
E                 "prefix": "translate English to Romanian: "
E               }
E             },
E             "transformers_version": "4.20.0.dev0",
E             "use_cache": true,
E             "vocab_size": 32128
E           }
E           
E           [INFO|tokenization_utils_base.py:1781] 2022-06-09 20:27:08,559 >> loading file https://huggingface.co/t5-small/resolve/main/spiece.model from cache at /root/.cache/huggingface/transformers/65fc04e21f45f61430aea0c4fedffac16a4d20d78b8e6601d8d996ebefefecd2.3b69006860e7b5d0a63ffdddc01ddcd6b7c318a6f4fd793596552c741734c62d
E           [INFO|tokenization_utils_base.py:1781] 2022-06-09 20:27:08,559 >> loading file https://huggingface.co/t5-small/resolve/main/tokenizer.json from cache at /root/.cache/huggingface/transformers/06779097c78e12f47ef67ecb728810c2ae757ee0a9efe9390c6419783d99382d.8627f1bd5d270a9fd2e5a51c8bec3223896587cc3cfe13edeabb0992ab43c529
E           [INFO|tokenization_utils_base.py:1781] 2022-06-09 20:27:08,559 >> loading file https://huggingface.co/t5-small/resolve/main/added_tokens.json from cache at None
E           [INFO|tokenization_utils_base.py:1781] 2022-06-09 20:27:08,559 >> loading file https://huggingface.co/t5-small/resolve/main/special_tokens_map.json from cache at None
E           [INFO|tokenization_utils_base.py:1781] 2022-06-09 20:27:08,559 >> loading file https://huggingface.co/t5-small/resolve/main/tokenizer_config.json from cache at None
E           [INFO|configuration_utils.py:659] 2022-06-09 20:27:08,700 >> loading configuration file https://huggingface.co/t5-small/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/fe501e8fd6425b8ec93df37767fcce78ce626e34cc5edc859c662350cf712e41.406701565c0afd9899544c1cb8b93185a76f00b31e5ce7f6e18bbaef02241985
E           [INFO|configuration_utils.py:708] 2022-06-09 20:27:08,701 >> Model config T5Config {
E             "_name_or_path": "t5-small",
E             "architectures": [
E               "T5WithLMHeadModel"
E             ],
E             "d_ff": 2048,
E             "d_kv": 64,
E             "d_model": 512,
E             "decoder_start_token_id": 0,
E             "dense_act_fn": "relu",
E             "dropout_rate": 0.1,
E             "eos_token_id": 1,
E             "feed_forward_proj": "relu",
E             "initializer_factor": 1.0,
E             "is_encoder_decoder": true,
E             "is_gated_act": false,
E             "layer_norm_epsilon": 1e-06,
E             "model_type": "t5",
E             "n_positions": 512,
E             "num_decoder_layers": 6,
E             "num_heads": 8,
E             "num_layers": 6,
E             "output_past": true,
E             "pad_token_id": 0,
E             "relative_attention_max_distance": 128,
E             "relative_attention_num_buckets": 32,
E             "task_specific_params": {
E               "summarization": {
E                 "early_stopping": true,
E                 "length_penalty": 2.0,
E                 "max_length": 200,
E                 "min_length": 30,
E                 "no_repeat_ngram_size": 3,
E                 "num_beams": 4,
E                 "prefix": "summarize: "
E               },
E               "translation_en_to_de": {
E                 "early_stopping": true,
E                 "max_length": 300,
E                 "num_beams": 4,
E                 "prefix": "translate English to German: "
E               },
E               "translation_en_to_fr": {
E                 "early_stopping": true,
E                 "max_length": 300,
E                 "num_beams": 4,
E                 "prefix": "translate English to French: "
E               },
E               "translation_en_to_ro": {
E                 "early_stopping": true,
E                 "max_length": 300,
E                 "num_beams": 4,
E                 "prefix": "translate English to Romanian: "
E               }
E             },
E             "transformers_version": "4.20.0.dev0",
E             "use_cache": true,
E             "vocab_size": 32128
E           }
E           
E           /workspace/transformers/src/transformers/models/t5/tokenization_t5_fast.py:156: FutureWarning: This tokenizer was incorrectly instantiated with a model max length of 512 which will be corrected in Transformers v5.
E           For now, this behavior is kept to avoid breaking backwards compatibility when padding/encoding with `truncation is True`.
E           - Be aware that you SHOULD NOT rely on t5-small automatically truncating your input to 512 when padding/encoding.
E           - If you want to encode/pad to sequences longer than 512 you can either instantiate this tokenizer with `model_max_length` or pass `max_length` when encoding/padding.
E           - To avoid this warning, please instantiate this tokenizer with `model_max_length` set to your preferred value.
E             warnings.warn(
E           [INFO|modeling_utils.py:2049] 2022-06-09 20:27:08,937 >> loading weights file https://huggingface.co/t5-small/resolve/main/pytorch_model.bin from cache at /root/.cache/huggingface/transformers/fee5a3a0ae379232608b6eed45d2d7a0d2966b9683728838412caccc41b4b0ed.ddacdc89ec88482db20c676f0861a336f3d0409f94748c209847b49529d73885
E           [INFO|modeling_utils.py:2425] 2022-06-09 20:27:09,933 >> All model checkpoint weights were used when initializing T5ForConditionalGeneration.
E           
E           [INFO|modeling_utils.py:2433] 2022-06-09 20:27:09,933 >> All the weights of T5ForConditionalGeneration were initialized from the model checkpoint at t5-small.
E           If your task is similar to the task the model of the checkpoint was trained on, you can already use T5ForConditionalGeneration for predictions without further training.
Running tokenizer on validation dataset: 100%|██████████| 1/1 [00:00<00:00, 60.72ba/s]
E           [INFO|trainer.py:519] 2022-06-09 20:27:10,499 >> Using cuda_amp half precision backend

src/transformers/testing_utils.py:1447: RuntimeError
```

The test wrapper gets in the way, you can test directly with the code the test is running (copy-n-pasted from your message):

```
deepspeed --num_nodes 1 --num_gpus 1 --master_port 10999 /workspace/transformers/examples/pytorch/translation/run_translation.py --model_name_or_path t5-small --train_file /workspace/transformers/tests/deepspeed/../fixtures/tests_samples/wmt_en_ro/train.json --validation_file /workspace/transformers/tests/deepspeed/../fixtures/tests_samples/wmt_en_ro/val.json --output_dir /tmp/tmpboxxylsc --overwrite_output_dir --max_source_length 32 --max_target_length 32 --val_max_target_length 32 --warmup_steps 8 --predict_with_generate --save_steps 0 --eval_steps 10 --group_by_length --label_smoothing_factor 0.1 --source_lang en --target_lang ro --report_to none --source_prefix "translate English to Romanian: " --fp16 --do_train --num_train_epochs 1 --max_train_samples 16 --per_device_train_batch_size 2 --learning_rate 3e-3 --do_eval --max_eval_samples 16 --per_device_eval_batch_size 2 --deepspeed /workspace
````

and then it should be easier to debug. In this case I think it's hanging and not returning anything hence there is no traceback from the sub-program.

And yes, `py-spy` is the next step to see where it's hanging.