Ray tune hyperparameters search error
Author: howardlau1999
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.1.0.dev0
- Platform: Linux-4.4.0-139-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

@sgugger

## Information

Model I am using (Bert, XLNet ...): Roberta-large

The problem arises when using:
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: GLUE SST-2

## To reproduce

Steps to reproduce the behavior:

1. I wanted to do a hyperparameter search so I referred to https://huggingface.co/blog/ray-tune and modified the `examples/text-classification/run_glue.py` replacing the training part with
```
    def model_init():
        model = AutoModelForSequenceClassification.from_pretrained(
            model_args.model_name_or_path,
            from_tf=bool(".ckpt" in model_args.model_name_or_path),
            config=config,
            cache_dir=model_args.cache_dir,
        )
        return model
    trainer = Trainer(
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset if training_args.do_eval else None,
        compute_metrics=compute_metrics,
        tokenizer=tokenizer,
        # Data collator will default to DataCollatorWithPadding, so we change it if we already did the padding.
        data_collator=default_data_collator if data_args.pad_to_max_length else None,
        model_init=model_init,
    )
```
```
    # Training
    if training_args.do_train:
        from ray import tune
        import ray
        ray.init()
        best_trial = trainer.hyperparameter_search(
            hp_space=lambda _ : {"seed": tune.grid_search([31, 42, 53])},
            direction="maximize", 
            backend="ray",
        )
        logger.info(" Best run %s" % str(best_trial))
```

2. Run `python run_glue.py --model_name_or_path roberta-large --do_train --do_eval --per_gpu_train_batch_size 8 --output_dir hypersearch-0 --task_name sst2 --evaluation_strategy steps --eval_steps 20 --logging_steps 10`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

Then the script exited with exception:

```
Traceback (most recent call last):
  File "run_glue.py", line 428, in <module>
    main()
  File "run_glue.py", line 359, in main
    best_trial = trainer.hyperparameter_search(
  File "/data1/howard/transformers/src/transformers/trainer.py", line 1039, in hyperparameter_search
    best_run = run_hp_search(self, n_trials, direction, **kwargs)
  File "/data1/howard/transformers/src/transformers/integrations.py", line 241, in run_hp_search_ray
    analysis = ray.tune.run(_objective, config=trainer.hp_space(None), num_samples=n_trials, **kwargs)
  File "/home/howard/anaconda3/envs/transformers/lib/python3.8/site-packages/ray/tune/tune.py", line 299, in run
    experiments[i] = Experiment(
  File "/home/howard/anaconda3/envs/transformers/lib/python3.8/site-packages/ray/tune/experiment.py", line 138, in __init__
    self._run_identifier = Experiment.register_if_needed(run)
  File "/home/howard/anaconda3/envs/transformers/lib/python3.8/site-packages/ray/tune/experiment.py", line 276, in register_if_needed
    register_trainable(name, run_object)
  File "/home/howard/anaconda3/envs/transformers/lib/python3.8/site-packages/ray/tune/registry.py", line 71, in register_trainable
    _global_registry.register(TRAINABLE_CLASS, name, trainable)
  File "/home/howard/anaconda3/envs/transformers/lib/python3.8/site-packages/ray/tune/registry.py", line 124, in register
    self.flush_values()
  File "/home/howard/anaconda3/envs/transformers/lib/python3.8/site-packages/ray/tune/registry.py", line 146, in flush_values
    _internal_kv_put(_make_key(category, key), value, overwrite=True)
  File "/home/howard/anaconda3/envs/transformers/lib/python3.8/site-packages/ray/experimental/internal_kv.py", line 27, in _internal_kv_put
    updated = worker.redis_client.hset(key, "value", value)
  File "/home/howard/anaconda3/envs/transformers/lib/python3.8/site-packages/redis/client.py", line 3004, in hset
    return self.execute_command('HSET', name, key, value)
  File "/home/howard/anaconda3/envs/transformers/lib/python3.8/site-packages/redis/client.py", line 877, in execute_command
    conn.send_command(*args)
  File "/home/howard/anaconda3/envs/transformers/lib/python3.8/site-packages/redis/connection.py", line 720, in send_command
    self.send_packed_command(self.pack_command(*args),
  File "/home/howard/anaconda3/envs/transformers/lib/python3.8/site-packages/redis/connection.py", line 712, in send_packed_command
    raise ConnectionError("Error %s while writing to socket. %s." %
redis.exceptions.ConnectionError: Error 104 while writing to socket. Connection reset by peer.
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

The script should run without errors.

## Related Issues

https://github.com/ray-project/ray/issues/2931

https://ray.readthedocs.io/en/latest/tune-usage.html#handling-large-datasets



Comments:
I googled for the error and it may be related to sending a large object to redis. Was it because the datasets are too large?

Hi! Did you try to open an issue at ray directly? It seems to be linked to their library rather than `transformers`

> Hi! Did you try to open an issue at ray directly? It seems to be linked to their library rather than `transformers`

I googled and found some related issues: https://github.com/ray-project/ray/issues/2931 and according to the replies the solution is https://ray.readthedocs.io/en/latest/tune-usage.html#handling-large-datasets

But I don't know how to pass that `tune.with_parameters`. Maybe the `Trainer` should take care of this?

It looks like something way too complex to implement so I'd suggest using optuna and see if you have the same problem, or re-implementing your own loop to use `ray.tune` on this. I don't think it can be supported easily by `Trainer`, and the documentation on the ray side is a bit too sparse on this subject to help us do it ourselves.

I have the same issue, and Optuna seems to be working fine. I think the biggest difference is that Optuna uses SQLite / in-memory, where Ray wants to send a (very large) object to Redis.

I don't have a solution for this problem, but just for others that might encounter the same problem, I tried the proposed solution (passing the arguments to `tune.run` via `ray.tune.with_parameters` in `run_hp_search_ray`) but the results were exactly the same. By what I have been able to gather, I would say that the problem arises from models bigger than 512M, not from the datasets.


hey folks, this should be working on the latest version of ray -- could you try installing the newest version via `pip install -U ray` and trying again?

> 
> 
> hey folks, this should be working on the latest version of ray -- could you try installing the newest version via `pip install -U ray` and trying again?

Hi @richardliaw! After updating ray to the latest version (1.1.0), it still isn't working for me, although the exception stack trace has changed a little (prior to this, I got the same exception as @howardlau1999 in their first comment):

```Traceback (most recent call last):
  File "/DATA/nperez/VENV/DNG/lib/python3.7/site-packages/redis/connection.py", line 706, in send_packed_command
    sendall(self._sock, item)
  File "/DATA/nperez/VENV/DNG/lib/python3.7/site-packages/redis/_compat.py", line 9, in sendall
    return sock.sendall(*args, **kwargs)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/DATA/nperez/PROJECTS/DNG/src/system/train_span_in_context.py", line 266, in <module>
    main()
  File "/DATA/nperez/PROJECTS/DNG/src/system/train_span_in_context.py", line 142, in main
    local_dir='/DATA/nperez/PROJECTS/DNG/hsearch/ray-search/'
  File "/DATA/nperez/VENV/DNG/lib/python3.7/site-packages/transformers/trainer.py", line 979, in hyperparameter_search
    best_run = run_hp_search(self, n_trials, direction, **kwargs)
  File "/DATA/nperez/VENV/DNG/lib/python3.7/site-packages/transformers/integrations.py", line 187, in run_hp_search_ray
    analysis = ray.tune.run(_objective, config=trainer.hp_space(None), num_samples=n_trials, **kwargs)
  File "/DATA/nperez/VENV/DNG/lib/python3.7/site-packages/ray/tune/tune.py", line 325, in run
    restore=restore)
  File "/DATA/nperez/VENV/DNG/lib/python3.7/site-packages/ray/tune/experiment.py", line 149, in __init__
    self._run_identifier = Experiment.register_if_needed(run)
  File "/DATA/nperez/VENV/DNG/lib/python3.7/site-packages/ray/tune/experiment.py", line 287, in register_if_needed
    register_trainable(name, run_object)
  File "/DATA/nperez/VENV/DNG/lib/python3.7/site-packages/ray/tune/registry.py", line 71, in register_trainable
    _global_registry.register(TRAINABLE_CLASS, name, trainable)
  File "/DATA/nperez/VENV/DNG/lib/python3.7/site-packages/ray/tune/registry.py", line 124, in register
    self.flush_values()
  File "/DATA/nperez/VENV/DNG/lib/python3.7/site-packages/ray/tune/registry.py", line 146, in flush_values
    _internal_kv_put(_make_key(category, key), value, overwrite=True)
  File "/DATA/nperez/VENV/DNG/lib/python3.7/site-packages/ray/experimental/internal_kv.py", line 27, in _internal_kv_put
    updated = worker.redis_client.hset(key, "value", value)
  File "/DATA/nperez/VENV/DNG/lib/python3.7/site-packages/redis/client.py", line 3050, in hset
    return self.execute_command('HSET', name, *items)
  File "/DATA/nperez/VENV/DNG/lib/python3.7/site-packages/redis/client.py", line 900, in execute_command
    conn.send_command(*args)
  File "/DATA/nperez/VENV/DNG/lib/python3.7/site-packages/redis/connection.py", line 726, in send_command
    check_health=kwargs.get('check_health', True))
  File "/DATA/nperez/VENV/DNG/lib/python3.7/site-packages/redis/connection.py", line 718, in send_packed_command
    (errno, errmsg))
redis.exceptions.ConnectionError: Error 32 while writing to socket. Broken pipe.
```

To be specific, in case it helps, I've _been_ able to make hyperparameter search work for the following pre-trained models&mdash;before and after updating ray&mdash;:
* dccuchile/bert-base-spanish-wwm-cased
* allenai/scibert_scivocab_cased
* skimai/spanberta-base-cased
* distilbert-base-uncased

But not these:
* bert-base-multilingual-cased
* xlm-roberta-base




I couldn't get ray tune working either for roberta-large after upgrading ray to version 1.1.0 @richardliaw

Got it! I'll take a closer look this week. Thanks!

Thanks for raising this issue. I could reproduce it (with `roberta-large`) on an AWS p2.xlarge instance. I created a PR that should fix this issue via `tune.with_parameters`: https://github.com/huggingface/transformers/pull/9749

@naiarapm it would be interesting to see what you did differently in your try to use `tune.with_parameters` - do you still have that piece of code available? We designed this utility exactly for handling large datasets and it worked for me in my experiments.

If you have the chance @howardlau1999 it would be great if you could check if this fixes your issue.

@krfricke Big thanks for your fix! I checked out your branch and the hyperparameters search with `ray` now works for me with `roberta-large`!

Hi @krfricke! 

Sorry for the delay. In response to your question, I simply changed the following line in `transformers.integrations.py`  (function `run_hp_search_ray`):

```
analysis = ray.tune.run(_objective, config=trainer.hp_space(None), num_samples=n_trials, **kwargs)
````

to this:

````
analysis = ray.tune.run(
    ray.tune.with_parameters(_objective),
    config=trainer.hp_space(None), num_samples=n_trials, **kwargs
)
````
I see now in your PR that that alone was not enough though :-) But I did not know what else to change, I just followed the suggested instructions to the best of my ability.

I can confirm as well that the error has been fixed for me. Thanks a lot!!