TypeError: can't pickle _thread.RLock objects hyperparameter_search raytune
Author: maxzzze
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: v4.5.1
- Platform: Linux
- Python version: 3.7.8
- PyTorch version (GPU?): yes
- Tensorflow version (GPU?):
- Using GPU in script?: 
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): bert-base-uncased 

The problem arises when using:
* [ x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run hyperparameter tuning with raytune
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

```
2021-04-14 15:44:01,389 INFO services.py:1264 -- View the Ray dashboard at http://127.0.0.1:8265
Traceback (most recent call last):
  File "pipeline_training.py", line 311, in <module>
    keep_checkpoints_num=0
  File "/opt/conda/lib/python3.7/site-packages/transformers/trainer.py", line 1459, in hyperparameter_search
    best_run = run_hp_search(self, n_trials, direction, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/transformers/integrations.py", line 235, in run_hp_search_ray
    **kwargs,
  File "/opt/conda/lib/python3.7/site-packages/ray/tune/tune.py", line 297, in run
    _ray_auto_init()
  File "/opt/conda/lib/python3.7/site-packages/ray/tune/tune.py", line 664, in _ray_auto_init
    ray.init()
  File "/opt/conda/lib/python3.7/site-packages/ray/_private/client_mode_hook.py", line 62, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/ray/worker.py", line 785, in init
    hook()
  File "/opt/conda/lib/python3.7/site-packages/ray/tune/registry.py", line 171, in flush
    self.references[k] = ray.put(v)
  File "/opt/conda/lib/python3.7/site-packages/ray/_private/client_mode_hook.py", line 62, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/ray/worker.py", line 1481, in put
    object_ref = worker.put_object(value)
  File "/opt/conda/lib/python3.7/site-packages/ray/worker.py", line 266, in put_object
    serialized_value = self.get_serialization_context().serialize(value)
  File "/opt/conda/lib/python3.7/site-packages/ray/serialization.py", line 324, in serialize
    return self._serialize_to_msgpack(value)
  File "/opt/conda/lib/python3.7/site-packages/ray/serialization.py", line 304, in _serialize_to_msgpack
    self._serialize_to_pickle5(metadata, python_objects)
  File "/opt/conda/lib/python3.7/site-packages/ray/serialization.py", line 264, in _serialize_to_pickle5
    raise e
  File "/opt/conda/lib/python3.7/site-packages/ray/serialization.py", line 261, in _serialize_to_pickle5
    value, protocol=5, buffer_callback=writer.buffer_callback)
  File "/opt/conda/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 73, in dumps
    cp.dump(obj)
  File "/opt/conda/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 580, in dump
    return Pickler.dump(self, obj)
TypeError: can't pickle _thread.RLock objects
```

The code chunk to start the `hyperparameter_search`: 

```python
def my_hp_space(trial):
    from ray import tune
    return {
        "learning_rate": tune.uniform(1e-5, 5e-5),
        "num_train_epochs": tune.choice(range(1, 6)),
        "per_device_train_batch_size": tune.choice([2,4]),
        "weight_decay": tune.uniform(0.0, 0.3),
        "adam_epsilon": tune.loguniform(1e-10, 1e-6),
        "per_device_eval_batch_size": 32
    }
    
best_run = trainer.hyperparameter_search(
    backend="ray",
    n_trials=15,
    hp_space=my_hp_space,
    stop=None,    
    checkpoint_score_attr="training_iteration",
    keep_checkpoints_num=0
    compute_objective=lambda x: my_objective(x, metric='eval_' + used_metric)
)
```

## Expected behavior

Expect that it will not throw an error. Note that this script does work on `4.2.0`. 

<!-- A clear and concise description of what you would expect to happen. -->



Comments:
I also have this issue (bump)

Pinging @richardliaw, @amogkam 

@maxzzze looks like a serialization error with the Trainer. We will take a look at this, but in the meantime can you downgrade your transformers version to 4.4. Also see https://github.com/ray-project/ray/issues/15439.

So it looks like this seems to work as soon as we disable the memory tracker:

```
trainer._memory_tracker = None
```

Will it be possible to expose an API to temporarily disable this? 

The other issue is https://github.com/huggingface/transformers/issues/11565, but we can resolve this there.

We should have tests that catch these regressions right?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I am having the same problem.
Disabling the memory tracker worked for me.
BUT, then I ran into #11565 as well

Yes, if you disable the memory tracker (pass in `skip_memory_metrics=True` into your `TrainingArguments`) then you will no longer get the pickling error.

In the next transformers release, the Ray Tune integration will automatically disable memory tracking if it's currently being enabled.

Hi, with transformers 4.26.1 on Sage maker I am still having this error: TypeError: cannot pickle '_thread.lock' object.

def hp_space(trial):
    return {
        "learning_rate": trial.suggest_float("learning_rate", 1e-5, 1e-3, log=True),
        "num_train_epochs": trial.suggest_int("num_train_epochs", 1, 10),
        "seed": trial.suggest_int("seed", 1, 40),
        "per_device_train_batch_size": trial.suggest_categorical("per_device_train_batch_size", [16, 32, 64]),
        "weight_decay": trial.suggest_float("weight_decay", 1e-3, 1e-1, log=True),
    }

best_run = trainer.hyperparameter_search(n_trials=20, direction="minimize", hp_space=hp_space)