text: 
  
  TITLE
  Deepspeed integration ignores Optuna trial parameters in hyperparameter_search
  
  COMMENTS
  4
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  ## Environment info
  <!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
       Don't forget to fill out the missing fields in that output! -->
  
  - `transformers` version: 4.6.1
  - Platform: Linux-4.19.0-16-cloud-amd64-x86_64-with-glibc2.10
  - Python version: 3.8.10
  - PyTorch version (GPU?): 1.8.1+cu111 (True)
  - Tensorflow version (GPU?): not installed (NA)
  - Using GPU in script?: yes
  - Using distributed or parallel set-up in script?: no
  
  Also in case it matters, my deepspeed version is 0.3.16.
  
  ### Who can help
  
  Maybe @stas00 ? I'm not sure.
  
  ## Information
  
  Model I am using: custom Pytorch transformer model, although I don't think it matters here.
  
  The problem arises when using:
  * [ ] the official example scripts: (probably, I haven't tried)
  * [x] my own modified scripts: I'm running a simple custom MLM training script using the transformers trainer.
  
  ## To reproduce
  
  Steps to reproduce the behavior:
  
  1. Run a hyperparameter search using the transformers Trainer with the [default zero-2 config from the documentation](https://huggingface.co/transformers/main_classes/trainer.html#zero-2-example)
  2. Observe that parameters taken from the Deepspeed config like gradient accumulation steps and optimizer/scheduler params are not updated to reflect the Optuna trial parameters.
  
  Here is some output from the script I'm running, with the middle omitted for brevity. I'm printing trial params myself but I do it from inside the Trainer, so these are definitely the same trial params the Trainer is getting.
  
  ```
  [I 2021-05-27 02:20:41,133] A new study created in memory with name: no-name-26248906-22c0-4666-a7d4-159173902bc5
  current trial params {'learning_rate': 2.0670100636747183e-05, 'adam_beta2': 0.98, 'gradient_accumulation_steps': 6, 'dropout': 0.0, 'local_attention_window': 192, 'weight_decay': 0.01, 'warmup_ratio': 0.12, 'deep_transformer_stack_layers': 10}
  
  .....
  
  [2021-05-27 02:21:01,787] [INFO] [config.py:751:print]   initial_dynamic_scale ........ 65536
  [2021-05-27 02:21:01,787] [INFO] [config.py:751:print]   loss_scale ................... 0
  [2021-05-27 02:21:01,787] [INFO] [config.py:751:print]   memory_breakdown ............. False
  [2021-05-27 02:21:01,787] [INFO] [config.py:751:print]   optimizer_legacy_fusion ...... False
  [2021-05-27 02:21:01,787] [INFO] [config.py:751:print]   optimizer_name ............... adamw
  [2021-05-27 02:21:01,787] [INFO] [config.py:751:print]   optimizer_params ............. {'lr': 5e-05, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.0}
  [2021-05-27 02:21:01,787] [INFO] [config.py:751:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}
  [2021-05-27 02:21:01,788] [INFO] [config.py:751:print]   pld_enabled .................. False
  
  ....
  ```
  I looked around briefly and I _think_ the issue comes from the fact that that the Deepspeed config [is built as part of TrainingArgs](https://github.com/huggingface/transformers/blob/996a315e76f6c972c854990e6114226a91bc0a90/src/transformers/training_args.py#L677) and then presumably never updated after that, even if the training args change. Consequently when the [training args are updated](https://github.com/huggingface/transformers/blob/996a315e76f6c972c854990e6114226a91bc0a90/src/transformers/trainer.py#L861) as part of setup for the hyperparameter search, it's not reflected in the Deepspeed config.
  
  Note that this might also be an issue with Ray, I just haven't tried it with Ray. 
  
  ## Expected behavior
  
  Ideally Deepspeed would run with config/parameters that respected the content of the Optuna trials, although I know that getting two external integrations to play well together is easier said than done. In the meantime I'm going to see if I can work around this by using an HF scheduler and HF optimizer in the hopes that those will take their parameters from the training arguments directly.
  
  