Cannot run grid search using Trainer API and Ray Tune
Author: mosh98
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.8.2
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.11
- PyTorch version (GPU?): 1.9.0+cu102 (True)
- Tensorflow version (GPU?): 2.6.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help

@richardliaw, @amogkam

@sgugger

<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.



Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- gpt2: @patrickvonplaten, @LysandreJik


Library:

- ray/raytune: @richardliaw, @amogkam

- trainer: @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): roBERTa

The problem arises when using:
* [x ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x ] my own task or dataset: (give details below)

## To reproduce

Hi, I am trying to do grid search on my roberta model.

Steps to reproduce the behavior:

1.

```
  hyperParameters = {
    "per_gpu_batch_size": [32],
    "learning_rate": [2e-5],
    "num_epochs": [2,3]
  }
  
```
```
  def my_hp_space_ray(trial):
      from ray import tune
  
      return {
          "learning_rate": tune.choice(hyperParameters.get('learning_rate')),
          "num_train_epochs": tune.choice(hyperParameters.get('num_epochs'))
      }
```
2.
```
training_args = TrainingArguments("test",
                                  per_device_train_batch_size= 32,
                                  per_device_eval_batch_size = 32,
                                  evaluation_strategy = "epoch", #Can be epoch or steps
                                  weight_decay=0.01,
                                  logging_strategy ="epoch",
                                  metric_for_best_model="accuracy",
                                  report_to="wandb"
                                  )
```
3.

```
trainer = Trainer(
    args=training_args,
    tokenizer=tokenizer,
    train_dataset=tokenized_datasets_train,
    eval_dataset=tokenized_datasets_val,
    model_init=model_init,
    compute_metrics=compute_metrics,
)
```

4.
```
trainer.hyperparameter_search(
    direction="minimize", 
    backend="ray",
    n_trials= 2,
    hp_space = my_hp_space_ray)
```



`
2021-08-30 21:07:06,743	ERROR trial_runner.py:773 -- Trial _objective_2e533_00001: Error processing event.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trial_runner.py", line 739, in _process_trial
    results = self.trial_executor.fetch_result(trial)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/ray_trial_executor.py", line 746, in fetch_result
    result = ray.get(trial_future[0], timeout=DEFAULT_GET_TIMEOUT)
  File "/usr/local/lib/python3.7/dist-packages/ray/_private/client_mode_hook.py", line 82, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/ray/worker.py", line 1621, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(TuneError): ray::ImplicitFunc.train_buffered() (pid=1182, ip=172.28.0.2, repr=<types.ImplicitFunc object at 0x7f67395b5ed0>)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trainable.py", line 178, in train_buffered
    result = self.train()
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trainable.py", line 237, in train
    result = self.step()
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 379, in step
    self._report_thread_runner_error(block=True)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 527, in _report_thread_runner_error
    ("Trial raised an exception. Traceback:\n{}".format(err_tb_str)
ray.tune.error.TuneError: Trial raised an exception. Traceback:
ray::ImplicitFunc.train_buffered() (pid=1182, ip=172.28.0.2, repr=<types.ImplicitFunc object at 0x7f67395b5ed0>)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 260, in run
    self._entrypoint()
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 329, in entrypoint
    self._status_reporter.get_checkpoint())
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 594, in _trainable_func
    output = fn()
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/utils/trainable.py", line 344, in inner
    trainable(config, **fn_kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/integrations.py", line 162, in _objective
    local_trainer.train(resume_from_checkpoint=checkpoint, trial=trial)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1269, in train
    tr_loss += self.training_step(model, inputs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1762, in training_step
    loss = self.compute_loss(model, inputs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1794, in compute_loss
    outputs = model(**inputs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py", line 1184, in forward
    return_dict=return_dict,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py", line 845, in forward
    return_dict=return_dict,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py", line 529, in forward
    output_attentions,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py", line 414, in forward
    past_key_value=self_attn_past_key_value,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py", line 344, in forward
    output_attentions,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py", line 257, in forward
    attention_scores = attention_scores / math.sqrt(self.attention_head_size)
RuntimeError: CUDA out of memory. Tried to allocate 384.00 MiB (GPU 0; 15.90 GiB total capacity; 13.14 GiB already allocated; 312.75 MiB free; 13.29 GiB reserved in total by PyTorch)
Result for _objective_2e533_00001:
  {}`
## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

Hi i would like to hyper parameter tune my roberta model, using ray tune and the trainer API, is there a way to not run out of memory even if it takes longer time to finish?  Or is there some other type of parameter tuning i should use instead?

I spent the whole day trying to figure it out, so any help would be hugely appreciated 

Comments:
Maybe try reducing the batch size of your model, do something like 8 or 4 first?

Tried with 4 and 8 i still get this error :( 

```
---------------------------------------------------------------------------
TuneError                                 Traceback (most recent call last)
<ipython-input-47-3b35fd348675> in <module>()
      3     backend="ray",
      4     n_trials= 1,
----> 5     hp_space = my_hp_space_ray
      6     )

2 frames
/usr/local/lib/python3.7/dist-packages/ray/tune/tune.py in run(run_or_experiment, name, metric, mode, stop, time_budget_s, config, resources_per_trial, num_samples, local_dir, search_alg, scheduler, keep_checkpoints_num, checkpoint_score_attr, checkpoint_freq, checkpoint_at_end, verbose, progress_reporter, log_to_file, trial_name_creator, trial_dirname_creator, sync_config, export_formats, max_failures, fail_fast, restore, server_port, resume, queue_trials, reuse_actors, trial_executor, raise_on_failed_trial, callbacks, loggers, ray_auto_init, run_errored_only, global_checkpoint_period, with_server, upload_dir, sync_to_cloud, sync_to_driver, sync_on_checkpoint, _remote)
    553     if incomplete_trials:
    554         if raise_on_failed_trial and not state[signal.SIGINT]:
--> 555             raise TuneError("Trials did not complete", incomplete_trials)
    556         else:
    557             logger.error("Trials did not complete: %s", incomplete_trials)

TuneError: ('Trials did not complete', [_objective_33435_00000])
```

Can you provide the full traceback?

On Tue, Aug 31, 2021 at 1:29 AM Mosleh Mahamud ***@***.***>
wrote:

> Tried with 4 and 8 i still get this error :(
>
> ---------------------------------------------------------------------------
> TuneError                                 Traceback (most recent call last)
> <ipython-input-47-3b35fd348675> in <module>()
>       3     backend="ray",
>       4     n_trials= 1,
> ----> 5     hp_space = my_hp_space_ray
>       6     )
>
> 2 frames
> /usr/local/lib/python3.7/dist-packages/ray/tune/tune.py in run(run_or_experiment, name, metric, mode, stop, time_budget_s, config, resources_per_trial, num_samples, local_dir, search_alg, scheduler, keep_checkpoints_num, checkpoint_score_attr, checkpoint_freq, checkpoint_at_end, verbose, progress_reporter, log_to_file, trial_name_creator, trial_dirname_creator, sync_config, export_formats, max_failures, fail_fast, restore, server_port, resume, queue_trials, reuse_actors, trial_executor, raise_on_failed_trial, callbacks, loggers, ray_auto_init, run_errored_only, global_checkpoint_period, with_server, upload_dir, sync_to_cloud, sync_to_driver, sync_on_checkpoint, _remote)
>     553     if incomplete_trials:
>     554         if raise_on_failed_trial and not state[signal.SIGINT]:
> --> 555             raise TuneError("Trials did not complete", incomplete_trials)
>     556         else:
>     557             logger.error("Trials did not complete: %s", incomplete_trials)
>
> TuneError: ('Trials did not complete', [_objective_33435_00000])
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/13348#issuecomment-909018861>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABCRZZK4FIVYD546RHIR2SDT7SHFZANCNFSM5DDPNGOA>
> .
>


yes here you go, seems like ray tune doesn't want to run at all :(
```

`No `resources_per_trial` arg was passed into `hyperparameter_search`. Setting it to a default value of 1 CPU and 1 GPU for each trial.
2021-08-31 08:49:52,166	WARNING callback.py:117 -- The TensorboardX logger cannot be instantiated because either TensorboardX or one of it's dependencies is not installed. Please make sure you have the latest version of TensorboardX installed: `pip install -U tensorboardx`
== Status ==
Memory usage on this node: 6.4/51.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/8 CPUs, 0/1 GPUs, 0.0/30.31 GiB heap, 0.0/15.15 GiB objects (0.0/1.0 accelerator_type:P100)
Result logdir: /root/ray_results/_objective_2021-08-31_08-49-52
Number of trials: 1/1 (1 PENDING)
+------------------------+----------+-------+-----------------+--------------------+
| Trial name             | status   | loc   |   learning_rate |   num_train_epochs |
|------------------------+----------+-------+-----------------+--------------------|
| _objective_680b5_00000 | PENDING  |       |           2e-05 |                  2 |
+------------------------+----------+-------+-----------------+--------------------+


(pid=1376) Some weights of the model checkpoint at roberta-base were not used when initializing RobertaForSequenceClassification: ['pooler.dense.weight', 'pooler.dense.bias']
(pid=1376) - This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
(pid=1376) - This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
(pid=1376) Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at roberta-base and are newly initialized: ['classifier.out_proj.weight', 'classifier.dense.weight', 'classifier.out_proj.bias', 'classifier.dense.bias']
(pid=1376) You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
(pid=1376) wandb: Currently logged in as: mosh98 (use `wandb login --relogin` to force relogin)
(pid=1376) wandb: Tracking run with wandb version 0.12.1
(pid=1376) wandb: Syncing run test
(pid=1376) wandb:  View project at https://wandb.ai/mosh98/Binary_Tuned
(pid=1376) wandb:  View run at https://wandb.ai/mosh98/Binary_Tuned/runs/15myily9
(pid=1376) wandb: Run data is saved locally in /root/ray_results/_objective_2021-08-31_08-49-52/_objective_680b5_00000_0_learning_rate=2e-05,num_train_epochs=2_2021-08-31_08-49-52/wandb/run-20210831_085001-15myily9
(pid=1376) wandb: Run `wandb offline` to turn off syncing.
(pid=1376) 
(pid=1376) signal only works in main thread
(pid=1376) <IPython.core.display.HTML object>
(pid=1376) <IPython.core.display.HTML object>
(pid=1376) <IPython.core.display.HTML object>
(pid=1376) <IPython.core.display.HTML object>
(pid=1376) <IPython.core.display.HTML object>
(pid=1376) <IPython.core.display.HTML object>
(pid=1376) <IPython.core.display.HTML object>
(pid=1376) <IPython.core.display.HTML object>
(pid=1376) 2021-08-31 08:50:10,695	ERROR function_runner.py:266 -- Runner Thread raised error.
(pid=1376) Traceback (most recent call last):
(pid=1376)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 260, in run
(pid=1376)     self._entrypoint()
(pid=1376)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 329, in entrypoint
(pid=1376)     self._status_reporter.get_checkpoint())
(pid=1376)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 594, in _trainable_func
(pid=1376)     output = fn()
(pid=1376)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/utils/trainable.py", line 344, in inner
(pid=1376)     trainable(config, **fn_kwargs)
(pid=1376)   File "/usr/local/lib/python3.7/dist-packages/transformers/integrations.py", line 162, in _objective
(pid=1376)     local_trainer.train(resume_from_checkpoint=checkpoint, trial=trial)
(pid=1376)   File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1331, in train
(pid=1376)     self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
(pid=1376)   File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1426, in _maybe_log_save_evaluate
(pid=1376)     metrics = self.evaluate()
(pid=1376)   File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 2031, in evaluate
(pid=1376)     metric_key_prefix=metric_key_prefix,
(pid=1376)   File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 2260, in evaluation_loop
(pid=1376)     metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, label_ids=all_labels))
(pid=1376)   File "<ipython-input-38-b8a033e8f995>", line 5, in compute_metrics
(pid=1376) NameError: name 'metric' is not defined
(pid=1376) Exception in thread Thread-2:
(pid=1376) Traceback (most recent call last):
(pid=1376)   File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
(pid=1376)     self.run()
(pid=1376)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 279, in run
(pid=1376)     raise e
(pid=1376)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 260, in run
(pid=1376)     self._entrypoint()
(pid=1376)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 329, in entrypoint
(pid=1376)     self._status_reporter.get_checkpoint())
(pid=1376)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 594, in _trainable_func
(pid=1376)     output = fn()
(pid=1376)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/utils/trainable.py", line 344, in inner
(pid=1376)     trainable(config, **fn_kwargs)
(pid=1376)   File "/usr/local/lib/python3.7/dist-packages/transformers/integrations.py", line 162, in _objective
(pid=1376)     local_trainer.train(resume_from_checkpoint=checkpoint, trial=trial)
(pid=1376)   File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1331, in train
(pid=1376)     self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
(pid=1376)   File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1426, in _maybe_log_save_evaluate
(pid=1376)     metrics = self.evaluate()
(pid=1376)   File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 2031, in evaluate
(pid=1376)     metric_key_prefix=metric_key_prefix,
(pid=1376)   File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 2260, in evaluation_loop
(pid=1376)     metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, label_ids=all_labels))
(pid=1376)   File "<ipython-input-38-b8a033e8f995>", line 5, in compute_metrics
(pid=1376) NameError: name 'metric' is not defined
(pid=1376) 
2021-08-31 08:50:10,898	ERROR trial_runner.py:773 -- Trial _objective_680b5_00000: Error processing event.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trial_runner.py", line 739, in _process_trial
    results = self.trial_executor.fetch_result(trial)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/ray_trial_executor.py", line 746, in fetch_result
    result = ray.get(trial_future[0], timeout=DEFAULT_GET_TIMEOUT)
  File "/usr/local/lib/python3.7/dist-packages/ray/_private/client_mode_hook.py", line 82, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/ray/worker.py", line 1621, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(TuneError): ray::ImplicitFunc.train_buffered() (pid=1376, ip=172.28.0.2, repr=<ray.tune.function_runner.ImplicitFunc object at 0x7f920a17d850>)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trainable.py", line 178, in train_buffered
    result = self.train()
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trainable.py", line 237, in train
    result = self.step()
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 379, in step
    self._report_thread_runner_error(block=True)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 527, in _report_thread_runner_error
    ("Trial raised an exception. Traceback:\n{}".format(err_tb_str)
ray.tune.error.TuneError: Trial raised an exception. Traceback:
ray::ImplicitFunc.train_buffered() (pid=1376, ip=172.28.0.2, repr=<ray.tune.function_runner.ImplicitFunc object at 0x7f920a17d850>)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 260, in run
    self._entrypoint()
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 329, in entrypoint
    self._status_reporter.get_checkpoint())
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 594, in _trainable_func
    output = fn()
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/utils/trainable.py", line 344, in inner
    trainable(config, **fn_kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/integrations.py", line 162, in _objective
    local_trainer.train(resume_from_checkpoint=checkpoint, trial=trial)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1331, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1426, in _maybe_log_save_evaluate
    metrics = self.evaluate()
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 2031, in evaluate
    metric_key_prefix=metric_key_prefix,
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 2260, in evaluation_loop
    metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, label_ids=all_labels))
  File "<ipython-input-38-b8a033e8f995>", line 5, in compute_metrics
NameError: name 'metric' is not defined
Result for _objective_680b5_00000:
  {}
  
== Status ==
Memory usage on this node: 7.9/51.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/8 CPUs, 0/1 GPUs, 0.0/30.31 GiB heap, 0.0/15.15 GiB objects (0.0/1.0 accelerator_type:P100)
Result logdir: /root/ray_results/_objective_2021-08-31_08-49-52
Number of trials: 1/1 (1 ERROR)
+------------------------+----------+-------+-----------------+--------------------+
| Trial name             | status   | loc   |   learning_rate |   num_train_epochs |
|------------------------+----------+-------+-----------------+--------------------|
| _objective_680b5_00000 | ERROR    |       |           2e-05 |                  2 |
+------------------------+----------+-------+-----------------+--------------------+
Number of errored trials: 1
+------------------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Trial name             |   # failures | error file                                                                                                                                     |
|------------------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------|
| _objective_680b5_00000 |            1 | /root/ray_results/_objective_2021-08-31_08-49-52/_objective_680b5_00000_0_learning_rate=2e-05,num_train_epochs=2_2021-08-31_08-49-52/error.txt |
+------------------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------+

== Status ==
Memory usage on this node: 7.9/51.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/8 CPUs, 0/1 GPUs, 0.0/30.31 GiB heap, 0.0/15.15 GiB objects (0.0/1.0 accelerator_type:P100)
Result logdir: /root/ray_results/_objective_2021-08-31_08-49-52
Number of trials: 1/1 (1 ERROR)
+------------------------+----------+-------+-----------------+--------------------+
| Trial name             | status   | loc   |   learning_rate |   num_train_epochs |
|------------------------+----------+-------+-----------------+--------------------|
| _objective_680b5_00000 | ERROR    |       |           2e-05 |                  2 |
+------------------------+----------+-------+-----------------+--------------------+
Number of errored trials: 1
+------------------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Trial name             |   # failures | error file                                                                                                                                     |
|------------------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------|
| _objective_680b5_00000 |            1 | /root/ray_results/_objective_2021-08-31_08-49-52/_objective_680b5_00000_0_learning_rate=2e-05,num_train_epochs=2_2021-08-31_08-49-52/error.txt |
+------------------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------+

---------------------------------------------------------------------------
TuneError                                 Traceback (most recent call last)
<ipython-input-58-3b35fd348675> in <module>()
      3     backend="ray",
      4     n_trials= 1,
----> 5     hp_space = my_hp_space_ray
      6     )

2 frames
/usr/local/lib/python3.7/dist-packages/transformers/trainer.py in hyperparameter_search(self, hp_space, compute_objective, n_trials, direction, backend, hp_name, **kwargs)
   1687 
   1688         run_hp_search = run_hp_search_optuna if backend == HPSearchBackend.OPTUNA else run_hp_search_ray
-> 1689         best_run = run_hp_search(self, n_trials, direction, **kwargs)
   1690 
   1691         self.hp_search_backend = None

/usr/local/lib/python3.7/dist-packages/transformers/integrations.py in run_hp_search_ray(trainer, n_trials, direction, **kwargs)
    243         config=trainer.hp_space(None),
    244         num_samples=n_trials,
--> 245         **kwargs,
    246     )
    247     best_trial = analysis.get_best_trial(metric="objective", mode=direction[:3])

/usr/local/lib/python3.7/dist-packages/ray/tune/tune.py in run(run_or_experiment, name, metric, mode, stop, time_budget_s, config, resources_per_trial, num_samples, local_dir, search_alg, scheduler, keep_checkpoints_num, checkpoint_score_attr, checkpoint_freq, checkpoint_at_end, verbose, progress_reporter, log_to_file, trial_name_creator, trial_dirname_creator, sync_config, export_formats, max_failures, fail_fast, restore, server_port, resume, queue_trials, reuse_actors, trial_executor, raise_on_failed_trial, callbacks, loggers, ray_auto_init, run_errored_only, global_checkpoint_period, with_server, upload_dir, sync_to_cloud, sync_to_driver, sync_on_checkpoint, _remote)
    553     if incomplete_trials:
    554         if raise_on_failed_trial and not state[signal.SIGINT]:
--> 555             raise TuneError("Trials did not complete", incomplete_trials)
    556         else:
    557             logger.error("Trials did not complete: %s", incomplete_trials)

TuneError: ('Trials did not complete', [_objective_680b5_00000])`
```

Hey @mosh98, the failures are due to this:  
```
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1331, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1426, in _maybe_log_save_evaluate
    metrics = self.evaluate()
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 2031, in evaluate
    metric_key_prefix=metric_key_prefix,
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 2260, in evaluation_loop
    metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, label_ids=all_labels))
  File "<ipython-input-38-b8a033e8f995>", line 5, in compute_metrics
NameError: name 'metric' is not defined
```

which looks is coming from the `compute_metrics` function that you are using. What does that function look like?

Hi, my function looks like this.

```
def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    predictions = predictions[:, 0]
    return metric.compute(predictions=predictions, references=labels)
```


Yes just fixed the metric problem, thank you @amogkam  i forgot to download the correct metric module from Hugginface.

However i am getting this error from pickle, even though i updated to pickle 5


```
2021-08-31 16:50:26,299	ERROR trial_runner.py:773 -- Trial _objective_87c7c_00000: Error processing event.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trial_runner.py", line 739, in _process_trial
    results = self.trial_executor.fetch_result(trial)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/ray_trial_executor.py", line 746, in fetch_result
    result = ray.get(trial_future[0], timeout=DEFAULT_GET_TIMEOUT)
  File "/usr/local/lib/python3.7/dist-packages/ray/_private/client_mode_hook.py", line 82, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/ray/worker.py", line 1621, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(TuneError): ray::ImplicitFunc.train_buffered() (pid=988, ip=172.28.0.2, repr=<types.ImplicitFunc object at 0x7f9c2d0bbed0>)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trainable.py", line 178, in train_buffered
    result = self.train()
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trainable.py", line 237, in train
    result = self.step()
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 379, in step
    self._report_thread_runner_error(block=True)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 527, in _report_thread_runner_error
    ("Trial raised an exception. Traceback:\n{}".format(err_tb_str)
ray.tune.error.TuneError: Trial raised an exception. Traceback:
ray::ImplicitFunc.train_buffered() (pid=988, ip=172.28.0.2, repr=<types.ImplicitFunc object at 0x7f9c2d0bbed0>)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 260, in run
    self._entrypoint()
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 329, in entrypoint
    self._status_reporter.get_checkpoint())
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 594, in _trainable_func
    output = fn()
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/utils/trainable.py", line 343, in inner
    fn_kwargs[k] = parameter_registry.get(prefix + k)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/registry.py", line 190, in get
    return ray.get(self.references[k])
ray.exceptions.RaySystemError: System error: No module named 'datasets_modules'
traceback: Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/ray/serialization.py", line 254, in deserialize_objects
    obj = self._deserialize_object(data, metadata, object_ref)
  File "/usr/local/lib/python3.7/dist-packages/ray/serialization.py", line 190, in _deserialize_object
    return self._deserialize_msgpack_data(data, metadata_fields)
  File "/usr/local/lib/python3.7/dist-packages/ray/serialization.py", line 168, in _deserialize_msgpack_data
    python_objects = self._deserialize_pickle5_data(pickle5_data)
  File "/usr/local/lib/python3.7/dist-packages/ray/serialization.py", line 156, in _deserialize_pickle5_data
    obj = pickle.loads(in_band, buffers=buffers)
ModuleNotFoundError: No module named 'datasets_modules'
(pid=988) 2021-08-31 16:50:26,242	ERROR serialization.py:256 -- No module named 'datasets_modules'
(pid=988) Traceback (most recent call last):
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/serialization.py", line 254, in deserialize_objects
(pid=988)     obj = self._deserialize_object(data, metadata, object_ref)
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/serialization.py", line 190, in _deserialize_object
(pid=988)     return self._deserialize_msgpack_data(data, metadata_fields)
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/serialization.py", line 168, in _deserialize_msgpack_data
(pid=988)     python_objects = self._deserialize_pickle5_data(pickle5_data)
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/serialization.py", line 156, in _deserialize_pickle5_data
(pid=988)     obj = pickle.loads(in_band, buffers=buffers)
(pid=988) ModuleNotFoundError: No module named 'datasets_modules'
(pid=988) 2021-08-31 16:50:26,242	ERROR function_runner.py:266 -- Runner Thread raised error.
(pid=988) Traceback (most recent call last):
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 260, in run
(pid=988)     self._entrypoint()
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 329, in entrypoint
(pid=988)     self._status_reporter.get_checkpoint())
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 594, in _trainable_func
(pid=988)     output = fn()
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/utils/trainable.py", line 343, in inner
(pid=988)     fn_kwargs[k] = parameter_registry.get(prefix + k)
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/registry.py", line 190, in get
(pid=988)     return ray.get(self.references[k])
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/_private/client_mode_hook.py", line 82, in wrapper
(pid=988)     return func(*args, **kwargs)
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/worker.py", line 1623, in get
(pid=988)     raise value
(pid=988) ray.exceptions.RaySystemError: System error: No module named 'datasets_modules'
(pid=988) traceback: Traceback (most recent call last):
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/serialization.py", line 254, in deserialize_objects
(pid=988)     obj = self._deserialize_object(data, metadata, object_ref)
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/serialization.py", line 190, in _deserialize_object
(pid=988)     return self._deserialize_msgpack_data(data, metadata_fields)
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/serialization.py", line 168, in _deserialize_msgpack_data
(pid=988)     python_objects = self._deserialize_pickle5_data(pickle5_data)
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/serialization.py", line 156, in _deserialize_pickle5_data
(pid=988)     obj = pickle.loads(in_band, buffers=buffers)
(pid=988) ModuleNotFoundError: No module named 'datasets_modules'
(pid=988) 
(pid=988) Exception in thread Thread-2:
(pid=988) Traceback (most recent call last):
(pid=988)   File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
(pid=988)     self.run()
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 279, in run
(pid=988)     raise e
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 260, in run
(pid=988)     self._entrypoint()
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 329, in entrypoint
(pid=988)     self._status_reporter.get_checkpoint())
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 594, in _trainable_func
(pid=988)     output = fn()
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/utils/trainable.py", line 343, in inner
(pid=988)     fn_kwargs[k] = parameter_registry.get(prefix + k)
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/registry.py", line 190, in get
(pid=988)     return ray.get(self.references[k])
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/_private/client_mode_hook.py", line 82, in wrapper
(pid=988)     return func(*args, **kwargs)
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/worker.py", line 1623, in get
(pid=988)     raise value
(pid=988) ray.exceptions.RaySystemError: System error: No module named 'datasets_modules'
(pid=988) traceback: Traceback (most recent call last):
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/serialization.py", line 254, in deserialize_objects
(pid=988)     obj = self._deserialize_object(data, metadata, object_ref)
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/serialization.py", line 190, in _deserialize_object
(pid=988)     return self._deserialize_msgpack_data(data, metadata_fields)
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/serialization.py", line 168, in _deserialize_msgpack_data
(pid=988)     python_objects = self._deserialize_pickle5_data(pickle5_data)
(pid=988)   File "/usr/local/lib/python3.7/dist-packages/ray/serialization.py", line 156, in _deserialize_pickle5_data
(pid=988)     obj = pickle.loads(in_band, buffers=buffers)
(pid=988) ModuleNotFoundError: No module named 'datasets_modules'
(pid=988) 
(pid=988) 
Result for _objective_87c7c_00000:
  {}
```

Seems like the error is:

```
(pid=1376)     output = fn()
(pid=1376)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/utils/trainable.py", line 344, in inner
(pid=1376)     trainable(config, **fn_kwargs)
(pid=1376)   File "/usr/local/lib/python3.7/dist-packages/transformers/integrations.py", line 162, in _objective
(pid=1376)     local_trainer.train(resume_from_checkpoint=checkpoint, trial=trial)
(pid=1376)   File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1331, in train
(pid=1376)     self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
(pid=1376)   File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1426, in _maybe_log_save_evaluate
(pid=1376)     metrics = self.evaluate()
(pid=1376)   File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 2031, in evaluate
(pid=1376)     metric_key_prefix=metric_key_prefix,
(pid=1376)   File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 2260, in evaluation_loop
(pid=1376)     metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, label_ids=all_labels))
(pid=1376)   File "<ipython-input-38-b8a033e8f995>", line 5, in compute_metrics
(pid=1376) NameError: name 'metric' is not defined
```

as a tip, maybe you could consider also doing `hyperparameter_search(..., fail_fast="raise")` to help see errors better.

Hmm, @Yard1 I thought there was a patch for the datasets_module thing?

There was - let me see

@mosh98 Hey can you try updating transformers? The patch to fix this issue was introduced in version v4.9.0 and you have said that you have v4.8.2

@Yard1 @richardliaw  updating transformers did help. Huge thanks!

altho i have one last question, is there a way to specify to use tpu's instead of gpu's in the api?

At the moment i have it like this,

```
trainer.hyperparameter_search(
    direction="minimize", 
    backend="ray",
    n_trials= 1,
    hp_space = my_hp_space_ray,
    resources_per_trial =  {"cpu": 1,"gpu": 1},
    fail_fast="raise"
    )
```

Hmm, great question; are you running this on Colab?

yes i am

do you guys recommend using ray tune for parameter optimization on cloud services rather than running on colab? @richardliaw @amogkam .

Sorry if i spamming you guys with question but i super new to ray tune and especially parameter optimization using Transformers.