[BUG] [Ray-Tune] ValueError: checkpoint not in list
Author: neel04
Pull request: no

## Environment info

- `transformers` version: 4.4.0.dev0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: Yes

### Who can help

Models:
- tensorflow: @jplu

Library:

- ray/raytune: @richardliaw, @amogkam

 -->

## Information

Model I am using (Bert, XLNet ...): RoBERTa

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

I want to do this as a text classification task where I have a sequence and I want to classify it into either one of the 20 labels (all of them numeric). 

Whenever I start tuning/Hyperparameter search, it starts running the first trial and logs a bit, then goes to the second trial showing the first one to be "running". This is how the logs look like:-
```

You are using PopulationBasedTraining but you haven't enabled checkpointing. This means your trials will train from scratch everytime they are exploiting new configurations. Consider enabling checkpointing by passing `keep_checkpoints_num=1` as an additional argument to `Trainer.hyperparameter_search`.

== Status ==
Memory usage on this node: 4.3/25.5 GiB
PopulationBasedTraining: 0 checkpoints, 0 perturbs
Resources requested: 3/4 CPUs, 1/1 GPUs, 0.0/14.99 GiB heap, 0.0/5.18 GiB objects (0/1.0 accelerator_type:P100)
Result logdir: /root/ray_results/tune_transformer_pbt
Number of trials: 1/100 (1 RUNNING)
+-----------------+----------+-------+-----------+-------+----------------+--------------+
| Trial name      | status   | loc   |   w_decay |    lr |   train_bs/gpu |   num_epochs |
|-----------------+----------+-------+-----------+-------+----------------+--------------|
| _inner_0755e982 | RUNNING  |       |  0.366291 | 4e-05 |              8 |           15 |
+-----------------+----------+-------+-----------+-------+----------------+--------------+


Result for _inner_0755e982:
  date: 2021-02-17_15-52-04
  done: false
  eval_accuracy: 0.14948453608247422
  eval_f1: 0.14948453608247422
  eval_loss: 8.17737865447998
  eval_precision: 0.14948453608247422
  eval_recall: 0.14948453608247422
  eval_runtime: 6.8976
  eval_samples_per_second: 56.252
  experiment_id: 5d2db84f7e9745a997bfcadedbd7d440
  hostname: 0df2f30fd76b
  iterations_since_restore: 1
  node_ip: 172.28.0.2
  objective: 0.14948453608247422
  pid: 39957
  time_since_restore: 21.89615297317505
  time_this_iter_s: 21.89615297317505
  time_total_s: 21.89615297317505
  timestamp: 1613577124
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 0755e982
  
== Status ==
Memory usage on this node: 6.3/25.5 GiB
PopulationBasedTraining: 0 checkpoints, 0 perturbs
Resources requested: 3/4 CPUs, 1/1 GPUs, 0.0/14.99 GiB heap, 0.0/5.18 GiB objects (0/1.0 accelerator_type:P100)
Result logdir: /root/ray_results/tune_transformer_pbt
Number of trials: 2/100 (1 PENDING, 1 RUNNING)
+-----------------+----------+------------------+-----------+-------+----------------+--------------+-------------+----------------------+
| Trial name      | status   | loc              |   w_decay |    lr |   train_bs/gpu |   num_epochs |   eval_loss |   training_iteration |
|-----------------+----------+------------------+-----------+-------+----------------+--------------+-------------+----------------------|
| _inner_0755e982 | RUNNING  | 172.28.0.2:39957 |  0.366291 | 4e-05 |              8 |           15 |     8.17738 |                    1 |
| _inner_07580d98 | PENDING  |                  |  0.376876 | 6e-05 |              8 |           10 |             |                      |
+-----------------+----------+------------------+-----------+-------+----------------+--------------+-------------+----------------------+


Result for _inner_0755e982:
  date: 2021-02-17_15-52-04
  done: false
  eval_accuracy: 0.14948453608247422
  eval_f1: 0.14948453608247422
  eval_loss: 8.17737865447998
  eval_precision: 0.14948453608247422
  eval_recall: 0.14948453608247422
  eval_runtime: 6.8976
  eval_samples_per_second: 56.252
  experiment_id: 5d2db84f7e9745a997bfcadedbd7d440
  experiment_tag: 1_num_train_epochs=15,per_device_eval_batch_size=16,per_device_train_batch_size=8
  hostname: 0df2f30fd76b
  iterations_since_restore: 1
  node_ip: 172.28.0.2
  objective: 0.14948453608247422
  pid: 39957
  time_since_restore: 21.89615297317505
  time_this_iter_s: 21.89615297317505
  time_total_s: 21.89615297317505
  timestamp: 1613577124
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 0755e982
  
Result for _inner_07580d98:
  date: 2021-02-17_15-52-29
  done: false
  eval_accuracy: 0.14948453608247422
  eval_f1: 0.14948453608247422
  eval_loss: 8.1666898727417
  eval_precision: 0.14948453608247422
  eval_recall: 0.14948453608247422
  eval_runtime: 6.8883
  eval_samples_per_second: 56.327
  experiment_id: e5cb4d5b00524454b7f673f971318b30
  hostname: 0df2f30fd76b
  iterations_since_restore: 1
  node_ip: 172.28.0.2
  objective: 0.14948453608247422
  pid: 39986
  time_since_restore: 21.889320135116577
  time_this_iter_s: 21.889320135116577
  time_total_s: 21.889320135116577
  timestamp: 1613577149
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 07580d98
  
== Status ==
Memory usage on this node: 6.3/25.5 GiB
PopulationBasedTraining: 0 checkpoints, 0 perturbs
Resources requested: 3/4 CPUs, 1/1 GPUs, 0.0/14.99 GiB heap, 0.0/5.18 GiB objects (0/1.0 accelerator_type:P100)
Result logdir: /root/ray_results/tune_transformer_pbt
Number of trials: 3/100 (1 ERROR, 1 PENDING, 1 RUNNING)
+-----------------+----------+------------------+-----------+-------+----------------+--------------+-------------+----------------------+
| Trial name      | status   | loc              |   w_decay |    lr |   train_bs/gpu |   num_epochs |   eval_loss |   training_iteration |
|-----------------+----------+------------------+-----------+-------+----------------+--------------+-------------+----------------------|
| _inner_07580d98 | RUNNING  | 172.28.0.2:39986 |  0.376876 | 6e-05 |              8 |           10 |     8.16669 |                    1 |
| _inner_15e144f6 | PENDING  |                  |  0.196785 | 2e-08 |              8 |           10 |             |                      |
| _inner_0755e982 | ERROR    |                  |  0.366291 | 4e-05 |              8 |           15 |     8.17738 |                    1 |
+-----------------+----------+------------------+-----------+-------+----------------+--------------+-------------+----------------------+
Number of errored trials: 1
+-----------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Trial name      |   # failures | error file                                                                                                                                                             |
|-----------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| _inner_0755e982 |            1 | /root/ray_results/tune_transformer_pbt/_inner_0755e982_1_num_train_epochs=15,per_device_eval_batch_size=16,per_device_train_batch_size=8_2021-02-17_15-51-42/error.txt |
+-----------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Result for _inner_07580d98:
  date: 2021-02-17_15-52-29
  done: false
  eval_accuracy: 0.14948453608247422
  eval_f1: 0.14948453608247422
  eval_loss: 8.1666898727417
  eval_precision: 0.14948453608247422
  eval_recall: 0.14948453608247422
  eval_runtime: 6.8883
  eval_samples_per_second: 56.327
  experiment_id: e5cb4d5b00524454b7f673f971318b30
  experiment_tag: 2_num_train_epochs=10,per_device_eval_batch_size=16,per_device_train_batch_size=8
  hostname: 0df2f30fd76b
  iterations_since_restore: 1
  node_ip: 172.28.0.2
  objective: 0.14948453608247422
  pid: 39986
  time_since_restore: 21.889320135116577
  time_this_iter_s: 21.889320135116577
  time_total_s: 21.889320135116577
  timestamp: 1613577149
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 07580d98
  
Result for _inner_15e144f6:
  date: 2021-02-17_15-52-53
  done: false
  eval_accuracy: 0.14948453608247422
  eval_f1: 0.14948453608247422
  eval_loss: 8.2146635055542
  eval_precision: 0.14948453608247422
  eval_recall: 0.14948453608247422
  eval_runtime: 6.917
  eval_samples_per_second: 56.094
  experiment_id: 2dc02378bb5d4a20a7c6d0228ad81076
  hostname: 0df2f30fd76b
  iterations_since_restore: 1
  node_ip: 172.28.0.2
  objective: 0.14948453608247422
  pid: 40016
  time_since_restore: 21.961735486984253
  time_this_iter_s: 21.961735486984253
  time_total_s: 21.961735486984253
  timestamp: 1613577173
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 15e144f6
  
== Status ==
Memory usage on this node: 6.2/25.5 GiB
PopulationBasedTraining: 0 checkpoints, 0 perturbs
Resources requested: 3/4 CPUs, 1/1 GPUs, 0.0/14.99 GiB heap, 0.0/5.18 GiB objects (0/1.0 accelerator_type:P100)
Result logdir: /root/ray_results/tune_transformer_pbt
Number of trials: 4/100 (2 ERROR, 1 PENDING, 1 RUNNING)
+-----------------+----------+------------------+-----------+-------+----------------+--------------+-------------+----------------------+
| Trial name      | status   | loc              |   w_decay |    lr |   train_bs/gpu |   num_epochs |   eval_loss |   training_iteration |
|-----------------+----------+------------------+-----------+-------+----------------+--------------+-------------+----------------------|
| _inner_15e144f6 | RUNNING  | 172.28.0.2:40016 |  0.196785 | 2e-08 |              8 |           10 |     8.21466 |                    1 |
| _inner_247daa04 | PENDING  |                  |  0.49907  | 5e-05 |              8 |           15 |             |                      |
| _inner_0755e982 | ERROR    |                  |  0.366291 | 4e-05 |              8 |           15 |     8.17738 |                    1 |
| _inner_07580d98 | ERROR    |                  |  0.376876 | 6e-05 |              8 |           10 |     8.16669 |                    1 |
+-----------------+----------+------------------+-----------+-------+----------------+--------------+-------------+----------------------+
Number of errored trials: 2
+-----------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Trial name      |   # failures | error file                                                                                                                                                             |
|-----------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| _inner_0755e982 |            1 | /root/ray_results/tune_transformer_pbt/_inner_0755e982_1_num_train_epochs=15,per_device_eval_batch_size=16,per_device_train_batch_size=8_2021-02-17_15-51-42/error.txt |
| _inner_07580d98 |            1 | /root/ray_results/tune_transformer_pbt/_inner_07580d98_2_num_train_epochs=10,per_device_eval_batch_size=16,per_device_train_batch_size=8_2021-02-17_15-52-06/error.txt |
+-----------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Result for _inner_15e144f6:
  date: 2021-02-17_15-52-53
  done: false
  eval_accuracy: 0.14948453608247422
  eval_f1: 0.14948453608247422
  eval_loss: 8.2146635055542
  eval_precision: 0.14948453608247422
  eval_recall: 0.14948453608247422
  eval_runtime: 6.917
  eval_samples_per_second: 56.094
  experiment_id: 2dc02378bb5d4a20a7c6d0228ad81076
  experiment_tag: 3_num_train_epochs=10,per_device_eval_batch_size=16,per_device_train_batch_size=8
  hostname: 0df2f30fd76b
  iterations_since_restore: 1
  node_ip: 172.28.0.2
  objective: 0.14948453608247422
  pid: 40016
  time_since_restore: 21.961735486984253
  time_this_iter_s: 21.961735486984253
  time_total_s: 21.961735486984253
  timestamp: 1613577173
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 15e144f6
```
And this is the error that usually pops up:-

```
2021-02-17 16:22:17,040	ERROR trial_runner.py:616 -- Trial _inner_34e77498: Error processing event.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/trial_runner.py", line 586, in _process_trial
    results = self.trial_executor.fetch_result(trial)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/ray_trial_executor.py", line 609, in fetch_result
    result = ray.get(trial_future[0], timeout=DEFAULT_GET_TIMEOUT)
  File "/usr/local/lib/python3.6/dist-packages/ray/_private/client_mode_hook.py", line 47, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/ray/worker.py", line 1456, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(TuneError): ray::ImplicitFunc.train_buffered() (pid=473, ip=172.28.0.2)
  File "python/ray/_raylet.pyx", line 480, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/trainable.py", line 167, in train_buffered
    result = self.train()
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/trainable.py", line 226, in train
    result = self.step()
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 366, in step
    self._report_thread_runner_error(block=True)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 513, in _report_thread_runner_error
    ("Trial raised an exception. Traceback:\n{}".format(err_tb_str)
ray.tune.error.TuneError: Trial raised an exception. Traceback:
ray::ImplicitFunc.train_buffered() (pid=473, ip=172.28.0.2)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 248, in run
    self._entrypoint()
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 316, in entrypoint
    self._status_reporter.get_checkpoint())
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 576, in _trainable_func
    output = fn()
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 651, in _inner
    inner(config, checkpoint_dir=None)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 645, in inner
    fn(config, **fn_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/integrations.py", line 160, in _objective
    local_trainer.train(resume_from_checkpoint=checkpoint, trial=trial)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 983, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1062, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1130, in _save_checkpoint
    self._rotate_checkpoints(use_mtime=True)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1460, in _rotate_checkpoints
    checkpoints_sorted = self._sorted_checkpoints(use_mtime=use_mtime)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1448, in _sorted_checkpoints
    best_model_index = checkpoints_sorted.index(str(Path(self.state.best_model_checkpoint)))
ValueError: 'results/run-34e77498/checkpoint-10' is not in list

```
Seems like it is trying to retrieve the best model but there is some sort of bug there (note that I am not checkpointing due to lack of storage capacity).

This is a part of the code that may have a clue as to the reason for this bug:-

```

from ray.tune.suggest.hyperopt import HyperOptSearch
from ray.tune.schedulers import PopulationBasedTraining
from ray.tune import CLIReporter, JupyterNotebookReporter
from ray import tune
import random

pbt = PopulationBasedTraining(
    time_attr="training_iteration",
    metric="eval_accuracy",
    mode="max",
    perturbation_interval=10,  # every 10 `time_attr` units
                               # (training_iterations in this case)
    hyperparam_mutations={

        "weight_decay": tune.uniform(1, 0.0001),
        "seed": tune.uniform(1,20000),
        "learning_rate": tune.choice([1e-5, 2e-5, 3e-5, 4e-5, 5e-5, 6e-5, 2e-7, 1e-7, 3e-7, 2e-8]),
        "adafactor": tune.choice(['True','False']),
        "adam_beta1": tune.uniform(1.0, 0.0),
        "adam_beta2": tune.uniform(1.0, 0),
        "adam_epsilon": tune.choice([1e-8, 2e-8, 3e-8, 1e-9, 2e-9, 3e-10]),
        "max_grad_norm": tune.uniform(1.0, 0),

    })


reporter = JupyterNotebookReporter(
    overwrite = True,
    metric = 'eval_accuracy',

    parameter_columns={
        "weight_decay": "w_decay",
        "learning_rate": "lr",
        "per_device_train_batch_size": "train_bs/gpu",
        "num_train_epochs": "num_epochs"},

    metric_columns=["eval_acc", "eval_loss", "epoch", "training_iteration"])

tune_config = {
        "per_device_train_batch_size": 8,
        "per_device_eval_batch_size": 16,
        "num_train_epochs": tune.choice([10,15])
        }

def compute_objective(metrics):
  return metrics["eval_accuracy"]
    
best = trainer.hyperparameter_search(hp_space = lambda _: tune_config, 
                                     n_trials=100, compute_objective=compute_objective, direction="maximize", backend='ray',    #search_alg=HyperOptSearch(metric='accuracy', mode='max', use_early_stopped_trials=True)
                                     scheduler=pbt, resources_per_trial={"cpu": 3, "gpu": 1}, keep_checkpoints_num=1,
                                     name = "tune_transformer_pbt", progress_reporter=reporter,
                                     search_alg=HyperOptSearch(metric='eval_accuracy', mode='max', use_early_stopped_trials=True),
                                     reuse_actors=True, checkpoint_at_end=True)
```
I can supply more code if requested, but it is more or less tweaked from official examples. 
So basically, the first trial keeps running, the second is put to pending while for the continuation of the rest of the trials, it results in an error (strangely not with the first one). 

I tried waiting till most trials have errored out to see whether it would continue to train the 1st trial but it just terminated giving the list of all trials that couldn't be completed.

> I am not sure where this bug is to be filed - rayproject or Huggingface, so I apologize in advance if I have posted in the wrong place.


Comments:
@neel04 can you try a few things:
- What version of Ray are you using? Can you try with the latest Ray (1.2).
- When using the PBT scheduler, it's actually not compatible with Tune search algorithms (see the compatibility matrix here https://docs.ray.io/en/master/tune/api_docs/schedulers.html#summary). Can you remove either HyperOptSearch or PopulationBasedTraining and try again.
- Can you pass in an absolute path to the `output_dir` in `TrainingArguments` instead of a relative one. I think right now it's being set to `./results`.
- And just make sure to run all cells in the notebook from scratch just in case any state is being saved from previous runs.

The error you posted is coming Huggingface checkpointing, so a person from HF might be better suited to help out here.

Also, if none of the above works for you, it would help if you could post a small, reproducible example, perhaps with dummy data, that can be run. Thanks!

- I am using the Latest Ray version: `1.2.0`
- Trying `PBT` alone without HyperOPT yields the same error. I always scrap the kernel after most runs due to this reason only, but it yields the same error on all configurations (12GB RAM or 25, P100 or V100).
- Set all paths to absolute, made no difference.

I have shared a [gist](https://colab.research.google.com/drive/1uuhCac9hTw1dcDnqpHuvZ63rMDTGNWWM?usp=sharing) that successfully reproduces the error with a dummy dataset. You can download the checkpoint `zip` [here in Google Drive](https://drive.google.com/drive/folders/1z8OgKtOxPlEDQV9905CyqyuaUEBx5FxK?usp=sharing).

It seems strange to me why `hyperparameter_search` in general is so buggy and complex :( as compared to solutions using native libraries.  

@neel04 thanks a lot for the posting this issue and the easy to run code. There is a bug with the checkpoint directories that is causing this error. This branch has a fix: https://github.com/amogkam/transformers/tree/tune-fix. Can you try it out and let me know if it is working for you. Thanks again!

@amogkam Thanx a lot for the branch. I am trying to use it but I am facing 2 problems:-

1. When constructing the `training_args` variable, It is not allowing me to use `adafactor` optimizer. Is this an expected limitation of this branch?
2. I am getting this error still :( that prevents all trials to run except the first one:-

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/trial_runner.py", line 586, in _process_trial
    results = self.trial_executor.fetch_result(trial)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/ray_trial_executor.py", line 609, in fetch_result
    result = ray.get(trial_future[0], timeout=DEFAULT_GET_TIMEOUT)
  File "/usr/local/lib/python3.6/dist-packages/ray/_private/client_mode_hook.py", line 47, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/ray/worker.py", line 1456, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(TuneError): ray::ImplicitFunc.train_buffered() (pid=1384, ip=172.28.0.2)
  File "python/ray/_raylet.pyx", line 480, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/trainable.py", line 167, in train_buffered
    result = self.train()
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/trainable.py", line 226, in train
    result = self.step()
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 366, in step
    self._report_thread_runner_error(block=True)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 513, in _report_thread_runner_error
    ("Trial raised an exception. Traceback:\n{}".format(err_tb_str)
ray.tune.error.TuneError: Trial raised an exception. Traceback:
ray::ImplicitFunc.train_buffered() (pid=1384, ip=172.28.0.2)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 248, in run
    self._entrypoint()
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 316, in entrypoint
    self._status_reporter.get_checkpoint())
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 576, in _trainable_func
    output = fn()
  File "/content/transformers/src/transformers/integrations.py", line 164, in _objective
    trainer.train(model_path=model_path, trial=trial)
  File "/content/transformers/src/transformers/trainer.py", line 757, in train
    for step, inputs in enumerate(epoch_iterator):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 475, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "<ipython-input-12-cd510628f360>", line 10, in __getitem__
TypeError: new(): invalid data type 'str'
```
Pretty huge tracebacks, but at least the previous error is gone which is an improvement. 
I will try to see if I can reproduce in the gist

**EDIT:-** scrap all that above. Right now, I am just focusing on the [gist](https://colab.research.google.com/drive/1uuhCac9hTw1dcDnqpHuvZ63rMDTGNWWM?usp=sharing) where I am still getting the Valueerror with a list. can you confirm that you can reproduce the error again?

@neel04 ah when installing from my fork you also have to specify the branch: `pip install git+https://github.com/amogkam/transformers.git@tune-fix`. The transformers version should be `4.4.0.dev0`. The gist works for me, and I'm not seeing the other 2 issues that you posted. Can you confirm that this branch works for you?

Yep, that fixes it :blush: Thanks for your help! one last thing - is it normal to see a single trial take a _very_ long time while having all the other trials paused?

@amogkam is this bug still present in 1.4.1? I am running into same problem when I set number of trails to more than 10.