0% GPU usage when using `hyperparameter_search`
Author: neel04
Pull request: no

_##_ Environment info

- `transformers` version: 4.4.0.dev0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No (Single GPU) --> Colab
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help

Models:
- ray/raytune: @richardliaw, @amogkam
- trainer: @sgugger

## Information

Model I am using (Bert, XLNet ...): RoBERTa

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

This is in continuation with #10055 where the underlying code is the same, and it is more or less the same as the official example. The problem is that when I start `hyperparameter_search` then it just keeps running with 0% GPU usage (memory is occupied) and the CPU also remains relatively idle:-
```

== Status ==
Memory usage on this node: 5.9/25.5 GiB
PopulationBasedTraining: 0 checkpoints, 0 perturbs
Resources requested: 1/4 CPUs, 1/1 GPUs, 0.0/14.99 GiB heap, 0.0/5.18 GiB objects (0/1.0 accelerator_type:P100)
Result logdir: /root/ray_results/_inner_2021-02-15_11-45-33
Number of trials: 1/100 (1 RUNNING)
+--------------------+----------+-------+-------------+--------------+--------------+----------------+-----------------+-----------------+--------------------+-------------------------------+---------+----------------+
| Trial name         | status   | loc   | adafactor   |   adam_beta1 |   adam_beta2 |   adam_epsilon |   learning_rate |   max_grad_norm |   num_train_epochs |   per_device_train_batch_size |    seed |   weight_decay |
|--------------------+----------+-------+-------------+--------------+--------------+----------------+-----------------+-----------------+--------------------+-------------------------------+---------+----------------|
| _inner_4fd43_00000 | RUNNING  |       | True        |     0.862131 |     0.813033 |          1e-09 |     2.34754e-05 |       0.0056821 |                  2 |                            16 | 21.1968 |        0.95152 |
+--------------------+----------+-------+-------------+--------------+--------------+----------------+-----------------+-----------------+--------------------+-------------------------------+---------+----------------+

```

Sometimes, there are also warnings that the single worker is pending due to lack of resources, however my CPU usage is minimum, plenty of RAM is free (~24 Gb) and GPU also some about a gig of free memory.
```

2021-02-15 13:56:53,761	WARNING worker.py:1107 -- The actor or task with ID ffffffffffffffff44ed5e1383be630817647ecd01000000 cannot be scheduled right now. It requires {CPU: 1.000000}, {GPU: 1.000000} for placement, but this node only has remaining {3.000000/4.000000 CPU, 14.990234 GiB/14.990234 GiB memory, 0.000000/1.000000 GPU, 1.000000/1.000000 node:172.28.0.2, 5.126953 GiB/5.126953 GiB object_store_memory, 1.000000/1.000000 accelerator_type:V100}
. In total there are 0 pending tasks and 1 pending actors on this node. This is likely due to all cluster resources being claimed by actors. To resolve the issue, consider creating fewer actors or increase the resources available to this Ray cluster. You can ignore this message if this Ray cluster is expected to auto-scale.
```

This is how the tuner looks like:-
```

from ray.tune.suggest.hyperopt import HyperOptSearch
from ray.tune.schedulers import PopulationBasedTraining
from ray import tune
import random

pbt = PopulationBasedTraining(
    time_attr="training_iteration",
    metric="accuracy",
    mode="max",
    perturbation_interval=10,  # every 10 `time_attr` units
                               # (training_iterations in this case)
    hyperparam_mutations={

        "weight_decay": tune.uniform(1, 0.0001),
        "seed": tune.uniform(1,20000),
        "learning_rate": tune.choice([1e-5, 2e-5, 3e-5, 4e-5, 5e-5, 6e-5, 2e-7, 1e-7, 3e-7, 2e-8]),
        "adafactor": tune.choice(['True','False']),
        "adam_beta1": tune.uniform(1.0, 0.0),
        "adam_beta2": tune.uniform(1.0, 0),
        "adam_epsilon": tune.choice([1e-8, 2e-8, 3e-8, 1e-9, 2e-9, 3e-10]),
        "max_grad_norm": tune.uniform(1.0, 0),

    })

best_run = trainer.hyperparameter_search(n_trials=100, compute_objective='accuracy', direction="maximize", backend='ray',
                                         scheduler=pbt)
```

Using `HyperOptScheduler` causes OOMs

Comments:
I tried decorating a function that contains the tranier command like this:-
```

@ray.remote(num_cpus=3, num_gpus=1, accelerator_type=ray.accelerators.NVIDIA_TESLA_V100)
def search():
  trainer.hyperparameter_search(n_trials=100, compute_objective='accuracy', direction="maximize", backend='ray',
                                scheduler=pbt)

search.remote()
```
but I am constantly getting:
`AttributeError: module 'ray' has no attribute 'accelerators'`
which I think is there because I may have written it the wrong way. can anyone shed any light on this?


Hi @neel04 can you add this arg to `trainer.hyperparameter_search`:

```
resources_per_trial={"cpu": 1, "gpu": 1}
```

This will let Tune know to reserve 1 CPU and 1 GPU for each trial.

Also, after instantiating your training_args, but before passing it into the `Trainer` can you also add this: `training_args._n_gpu = 1`.

Here is a more up to date example if you want to try it out and see if it works for you: https://github.com/amogkam/ray/blob/hf-pbt/python/ray/tune/examples/pbt_transformers/pbt_transformers.py


@amogkam Bless you sir! I am now getting mostly 100% GPU usage (but only around 6-7GB VRAM usage out of available 16Gb).
However, each of my trial is failing with this error:-
```

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/trial_runner.py", line 586, in _process_trial
    results = self.trial_executor.fetch_result(trial)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/ray_trial_executor.py", line 609, in fetch_result
    result = ray.get(trial_future[0], timeout=DEFAULT_GET_TIMEOUT)
  File "/usr/local/lib/python3.6/dist-packages/ray/_private/client_mode_hook.py", line 47, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/ray/worker.py", line 1456, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(TuneError): ray::ImplicitFunc.train_buffered() (pid=415, ip=172.28.0.2)
  File "python/ray/_raylet.pyx", line 480, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/trainable.py", line 167, in train_buffered
    result = self.train()
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/trainable.py", line 226, in train
    result = self.step()
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 366, in step
    self._report_thread_runner_error(block=True)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 513, in _report_thread_runner_error
    ("Trial raised an exception. Traceback:\n{}".format(err_tb_str)
ray.tune.error.TuneError: Trial raised an exception. Traceback:
ray::ImplicitFunc.train_buffered() (pid=415, ip=172.28.0.2)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 248, in run
    self._entrypoint()
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 316, in entrypoint
    self._status_reporter.get_checkpoint())
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 576, in _trainable_func
    output = fn()
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 651, in _inner
    inner(config, checkpoint_dir=None)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 645, in inner
    fn(config, **fn_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/integrations.py", line 160, in _objective
    local_trainer.train(resume_from_checkpoint=checkpoint, trial=trial)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 983, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1059, in _maybe_log_save_evaluate
    self._report_to_hp_search(trial, epoch, metrics)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 640, in _report_to_hp_search
    self.objective = self.compute_objective(metrics.copy())
TypeError: 'str' object is not callable
```
Is it something related to `ray/tune` or is this a wrong argument on my part?
BTW would you also happen to know how to set a fixed batch size for all trials? for some reason, it is overriding the `batch_size` provided in the Trainer_arguments and trying out it's own random ones

Ah this is because you are passing in `'accuracy'` as the `compute_objective` to `hyperparameter_search`. The `compute_objective` should actually be a function that computes the objective to minimize or maximize from the metrics returned by `evaluate`. You can also not pass one in, and it will default to `trainer_utils.default_compute_objective`. 

@amogkam MIssed that :( thanx a lot for taking the time out to help me!! :+1: :100: :1st_place_medal: 
Thanx to amogkam's comment, the issuse described here has been resolved, so I am closing this. but it is still giving the error:-
```

2021-02-15 18:58:56,343	ERROR worker.py:1053 -- Possible unhandled error from worker: ray::ImplicitFunc.train_buffered() (pid=2319, ip=172.28.0.2)
  File "python/ray/_raylet.pyx", line 480, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/trainable.py", line 167, in train_buffered
    result = self.train()
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/trainable.py", line 226, in train
    result = self.step()
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 366, in step
    self._report_thread_runner_error(block=True)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 513, in _report_thread_runner_error
    ("Trial raised an exception. Traceback:\n{}".format(err_tb_str)
ray.tune.error.TuneError: Trial raised an exception. Traceback:
ray::ImplicitFunc.train_buffered() (pid=2319, ip=172.28.0.2)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 248, in run
    self._entrypoint()
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 316, in entrypoint
    self._status_reporter.get_checkpoint())
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 576, in _trainable_func
    output = fn()
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 651, in _inner
    inner(config, checkpoint_dir=None)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 645, in inner
    fn(config, **fn_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/integrations.py", line 160, in _objective
    local_trainer.train(resume_from_checkpoint=checkpoint, trial=trial)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 983, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1059, in _maybe_log_save_evaluate
    self._report_to_hp_search(trial, epoch, metrics)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 640, in _report_to_hp_search
    self.objective = self.compute_objective(metrics.copy())
  File "<ipython-input-20-4880cfa49052>", line 5, in compute_metrics
AttributeError: 'dict' object has no attribute 'label_ids'
```
It's not clear to me why there is a whole string of errors on code copy/pasted from training models (which executes successfully). but there are 2 issues:-

1. Firstly, the `train_bs` is randomly selected and almost always causes OOM's  - arguments do not override this behavior
2. The above "dict" error happens, if OOM does not get it.

can anyone also explain why the method in the official docs doesn't work even though it is the same task with the same libraries? Has the `ray` framework changed and the docs do not reflect it? Honestly, it is becoming difficult exactly where to post these errors - ray OR HuggingFace. They require a person intimately familiar with both.

@neel04 are you running this example- https://github.com/amogkam/ray/blob/hf-pbt/python/ray/tune/examples/pbt_transformers/pbt_transformers.py? This is the most up to date one and should work with transformers v4.

I think the example is a bit verbose and some of it goes over my head :) So I am having difficulty in identifying what steps I have configured wrong and what exactly needs to be corrected. Understanding things like passing the `tune_configs` are easy to get, but the errors are much more difficult to track

OK got it. The stack trace you just posted is coming from the `compute_metrics` that is passed into your `Trainer`. What does that look like?

`compute_metrics` is standard code from the official example:-
```
from sklearn.metrics import accuracy_score, precision_recall_fscore_support

def compute_metrics(pred):
    labels = pred.label_ids
    preds = pred.predictions.argmax(-1)
    precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='weighted')  #none gives score for each class
    acc = accuracy_score(labels, preds)
    return {
        'accuracy': acc,
        'f1': f1,
        'precision': precision,
        'recall': recall
    }
```
It worked perfectly when training the model without any tuning, so I doubt the true error originates from here.
Also, would you mind telling me how you tracked the problem to `compute_metrics` since I couldn't sniff any clues to there in the error?

The stack trace says `line 5, in compute_metrics` so I thought it was coming from there. Do you mind sharing your full code to reproduce this?

Sure. Sorry if the code is a bit verbose.
```

%%capture
!pip install ray[tune]
!pip install ray
!pip install -q transformers

from sklearn.metrics import accuracy_score, precision_recall_fscore_support
from transformers import RobertaForSequenceClassification, Trainer, TrainingArguments

def compute_metrics(pred):
    labels = pred.label_ids
    preds = pred.predictions.argmax(-1)
    precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='weighted')  #none gives score for each class
    acc = accuracy_score(labels, preds)
    return {
        'accuracy': acc,
        'f1': f1,
        'precision': precision,
        'recall': recall
    }

training_args = TrainingArguments(
    output_dir='./results',          # output directory
    overwrite_output_dir = True,
    num_train_epochs=20,              # total number of training epochs
    per_device_train_batch_size=8,  # batch size per device during training
    per_device_eval_batch_size=8,   # batch size for evaluation,
    warmup_steps=500,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='./logs',            # directory for storing logs
    logging_steps=10,
    evaluation_strategy='steps',
    learning_rate=2e-5,
    fp16 = True,
    load_best_model_at_end = True,
    metric_for_best_model = 'accuracy',
    greater_is_better = True,
    seed = 101,
    do_eval = True,
    do_train = True,
    adam_beta1=0.9,
    adam_beta2=0.999,
    adam_epsilon=1e-8,
    max_grad_norm=1.0,
    adafactor = False

)

training_args._n_gpu = 1

def model_init():
    return RobertaForSequenceClassification.from_pretrained('/content/drive/MyDrive/checkpoint-2700', num_labels=20)
trainer = Trainer(
    model_init=model_init,
    args=training_args,
    train_dataset=train_dataset,       # Indexing
    eval_dataset=val_dataset,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics)

from ray.tune.suggest.hyperopt import HyperOptSearch
from ray.tune.schedulers import PopulationBasedTraining
from ray.tune import CLIReporter
from ray import tune
import random

pbt = PopulationBasedTraining(
    time_attr="training_iteration",
    metric="accuracy",
    mode="max",
    perturbation_interval=10,  # every 10 `time_attr` units
                               # (training_iterations in this case)
    hyperparam_mutations={

        "weight_decay": tune.uniform(1, 0.0001),
        "seed": tune.uniform(1,20000),
        "learning_rate": tune.choice([1e-5, 2e-5, 3e-5, 4e-5, 5e-5, 6e-5, 2e-7, 1e-7, 3e-7, 2e-8]),
        "adafactor": tune.choice(['True','False']),
        "adam_beta1": tune.uniform(1.0, 0.0),
        "adam_beta2": tune.uniform(1.0, 0),
        "adam_epsilon": tune.choice([1e-8, 2e-8, 3e-8, 1e-9, 2e-9, 3e-10]),
        "max_grad_norm": tune.uniform(1.0, 0),

    })


reporter = CLIReporter(
    parameter_columns={
        "weight_decay": "w_decay",
        "learning_rate": "lr",
        "per_device_train_batch_size": "train_bs/gpu",
        "num_train_epochs": "num_epochs"},

    metric_columns=["eval_acc", "eval_loss", "epoch", "training_iteration"])

tune_config = {
        "per_device_train_batch_size": 8,
        "per_device_eval_batch_size": 16,
        "num_train_epochs": tune.choice([15,20,25])
        }
    
best = trainer.hyperparameter_search(hp_space = lambda _: tune_config, 
                                     n_trials=100, compute_objective=compute_metrics, direction="maximize", backend='ray',    #search_alg=HyperOptSearch(metric='accuracy', mode='max', use_early_stopped_trials=True)
                                     scheduler=pbt, resources_per_trial={"cpu": 2, "gpu": 1}, keep_checkpoints_num=1,
                                     name = "tune_transformer_pbt", progress_reporter=reporter)
```

Ah @neel04, the error message is happening because `compute_metrics` is being passed as the `compute_objective` arg in `trainer.hyperparameter_search`. If you remove this arg your code runs fine.

`compute_objective` should be a function that takes in the output of `evaluate` (which is the dictionary returned `compute_metrics`) as an input and returns a single float value (see the docstring). It is not the same as `compute_metrics`. So here you should just be returning the "accuracy" value from the input dictionary. Something like this should work I believe:
```
def compute_objective(metrics):
  return metrics["accuracy"]
```

So I tried that above, but apparently `evaluate` does not return "accuracy", so as a workaround I switched to `eval_accuracy`.
But this creates a new problem; this error comes in the first trial **but** it doesn't go on to the next trial. Could be that it is training? GPU usage seems to be 0, so I doubt it is training but it is not terminating the process or moving on. Strange.
```

2021-02-17 10:57:12,244	ERROR worker.py:1053 -- Possible unhandled error from worker: ray::ImplicitFunc.train_buffered() (pid=1340, ip=172.28.0.2)
  File "python/ray/_raylet.pyx", line 480, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/trainable.py", line 167, in train_buffered
    result = self.train()
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/trainable.py", line 226, in train
    result = self.step()
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 366, in step
    self._report_thread_runner_error(block=True)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 513, in _report_thread_runner_error
    ("Trial raised an exception. Traceback:\n{}".format(err_tb_str)
ray.tune.error.TuneError: Trial raised an exception. Traceback:
ray::ImplicitFunc.train_buffered() (pid=1340, ip=172.28.0.2)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 248, in run
    self._entrypoint()
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 316, in entrypoint
    self._status_reporter.get_checkpoint())
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 576, in _trainable_func
    output = fn()
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 651, in _inner
    inner(config, checkpoint_dir=None)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 645, in inner
    fn(config, **fn_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/integrations.py", line 160, in _objective
    local_trainer.train(resume_from_checkpoint=checkpoint, trial=trial)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 925, in train
    for step, inputs in enumerate(epoch_iterator):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 475, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "<ipython-input-12-cd510628f360>", line 10, in __getitem__
TypeError: new(): invalid data type 'str'
```

It looks like it Is pointing to 'objective', which is the same function you wrote above:-

```
def compute_objective(metrics):
  return metrics["eval_accuracy"]           #does not return accuracy
```
Interestingly, removing the args `compute_objective`and `direction` does not yield anything, so I figured the problem must be elsewhere. 

Putting `eval_accuracy` in the PBT parameters and making the `compute_objective` solves the issue.

Thanks a lot @amogkam for your support!! we need more people like you :+1: :rocket: :partying_face: