Issue 1:
Environment info

transformers version: 4.6.1

Platform: Linux-4.19.0-16-cloud-amd64-x86_64-with-glibc2.10

Python version: 3.8.10

PyTorch version (GPU?): 1.8.1+cu111 (True)

Tensorflow version (GPU?): not installed (NA)

Using GPU in script?: yes

Using distributed or parallel set-up in script?: no

Also in case it matters, my deepspeed version is 0.3.16.

Who can help

Maybe @stas00 ? I'm not sure.

Information

Model I am using: custom Pytorch transformer model, although I don't think it matters here.

The problem arises when using:

the official example scripts: (probably, I haven't tried)

my own modified scripts: I'm running a simple custom MLM training script using the transformers trainer.

To reproduce

Steps to reproduce the behavior:

Run a hyperparameter search using the transformers Trainer with the default zero-2 config from the documentation

Observe that parameters taken from the Deepspeed config like gradient accumulation steps and optimizer/scheduler params are not updated to reflect the Optuna trial parameters.

Here is some output from the script I'm running, with the middle omitted for brevity. I'm printing trial params myself but I do it from inside the Trainer, so these are definitely the same trial params the Trainer is getting.

[I 2021-05-27 02:20:41,133] A new study created in memory with name: no-name-26248906-22c0-4666-a7d4-159173902bc5
current trial params {'learning_rate': 2.0670100636747183e-05, 'adam_beta2': 0.98, 'gradient_accumulation_steps': 6, 'dropout': 0.0, 'local_attention_window': 192, 'weight_decay': 0.01, 'warmup_ratio': 0.12, 'deep_transformer_stack_layers': 10}

.....

[2021-05-27 02:21:01,787] [INFO] [config.py:751:print] initial_dynamic_scale ........ 65536
[2021-05-27 02:21:01,787] [INFO] [config.py:751:print] loss_scale ................... 0
[2021-05-27 02:21:01,787] [INFO] [config.py:751:print] memory_breakdown ............. False
[2021-05-27 02:21:01,787] [INFO] [config.py:751:print] optimizer_legacy_fusion ...... False
[2021-05-27 02:21:01,787] [INFO] [config.py:751:print] optimizer_name ............... adamw
[2021-05-27 02:21:01,787] [INFO] [config.py:751:print] optimizer_params ............. {'lr': 5e-05, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.0}
[2021-05-27 02:21:01,787] [INFO] [config.py:751:print] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}
[2021-05-27 02:21:01,788] [INFO] [config.py:751:print] pld_enabled .................. False

....

I looked around briefly and I think the issue comes from the fact that that the Deepspeed config is built as part of TrainingArgs and then presumably never updated after that, even if the training args change. Consequently when the training args are updated as part of setup for the hyperparameter search, it's not reflected in the Deepspeed config.

Note that this might also be an issue with Ray, I just haven't tried it with Ray.

Expected behavior

Ideally Deepspeed would run with config/parameters that respected the content of the Optuna trials, although I know that getting two external integrations to play well together is easier said than done. In the meantime I'm going to see if I can work around this by using an HF scheduler and HF optimizer in the hopes that those will take their parameters from the training arguments directly.

Issue 2:
❓ Questions & Help

I am using GPT2Tokenizer. I observed that some tokens are duplicated in the vocabulary with a space appended to them in the beginning. For example, there exist two separate tokens - can and <space>can. Both of them are mapped to different token IDs - 5171 and 460 respectively.

Although, both -

can_token_id = tokenizer.encode(' can', add_special_tokens=False)

and

can_token_id = tokenizer.encode('can', add_special_tokens=False)

return can_token_id as [5171]

Is this an intended property of the tokenizer? Is there a mapping available between what token IDs are just a space appended version of other tokenIDs?