Repository: huggingface/transformers

Issue:
TypeError: can't pickle _thread.RLock objects hyperparameter_search raytune
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: v4.5.1
- Platform: Linux
- Python version: 3.7.8
- PyTorch version (GPU?): yes
- Tensorflow version (GPU?):
- Using GPU in script?: 
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): bert-base-uncased 

The problem arises when using:
* [ x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run hyperparameter tuning with raytune
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

```
2021-04-14 15:44:01,389 INFO services.py:1264 -- View the Ray dashboard at http://127.0.0.1:8265
Traceback (most recent call last):
  File "pipeline_training.py", line 311, in <module>
    keep_checkpoints_num=0
  File "/opt/conda/lib/python3.7/site-packages/transformers/trainer.py", line 1459, in hyperparameter_search
    best_run = run_hp_search(self, n_trials, direction, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/transformers/integrations.py", line 235, in run_hp_search_ray
    **kwargs,
  File "/opt/conda/lib/python3.7/site-packages/ray/tune/tune.py", line 297, in run
    _ray_auto_init()
  File "/opt/conda/lib/python3.7/site-packages/ray/tune/tune.py", line 664, in _ray_auto_init
    ray.init()
  File "/opt/conda/lib/python3.7/site-packages/ray/_private/client_mode_hook.py", line 62, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/ray/worker.py", line 785, in init
    hook()
  File "/opt/conda/lib/python3.7/site-packages/ray/tune/registry.py", line 171, in flush
    self.references[k] = ray.put(v)
  File "/opt/conda/lib/python3.7/site-packages/ray/_private/client_mode_hook.py", line 62, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/ray/worker.py", line 1481, in put
    object_ref = worker.put_object(value)
  File "/opt/conda/lib/python3.7/site-packages/ray/worker.py", line 266, in put_object
    serialized_value = self.get_serialization_context().serialize(value)
  File "/opt/conda/lib/python3.7/site-packages/ray/serialization.py", line 324, in serialize
    return self._serialize_to_msgpack(value)
  File "/opt/conda/lib/python3.7/site-packages/ray/serialization.py", line 304, in _serialize_to_msgpack
    self._serialize_to_pickle5(metadata, python_objects)
  File "/opt/conda/lib/python3.7/site-packages/ray/serialization.py", line 264, in _serialize_to_pickle5
    raise e
  File "/opt/conda/lib/python3.7/site-packages/ray/serialization.py", line 261, in _serialize_to_pickle5
    value, protocol=5, buffer_callback=writer.buffer_callback)
  File "/opt/conda/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 73, in dumps
    cp.dump(obj)
  File "/opt/conda/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 580, in dump
    return Pickler.dump(self, obj)
TypeError: can't pickle _thread.RLock objects
```

The code chunk to start the `hyperparameter_search`: 

```python
def my_hp_space(trial):
    from ray import tune
    return {
        "learning_rate": tune.uniform(1e-5, 5e-5),
        "num_train_epochs": tune.choice(range(1, 6)),
        "per_device_train_batch_size": tune.choice([2,4]),
        "weight_decay": tune.uniform(0.0, 0.3),
        "adam_epsilon": tune.loguniform(1e-10, 1e-6),
        "per_device_eval_batch_size": 32
    }
    
best_run = trainer.hyperparameter_search(
    backend="ray",
    n_trials=15,
    hp_space=my_hp_space,
    stop=None,    
    checkpoint_score_attr="training_iteration",
    keep_checkpoints_num=0
    compute_objective=lambda x: my_objective(x, metric='eval_' + used_metric)
)
```

## Expected behavior

Expect that it will not throw an error. Note that this script does work on `4.2.0`. 

<!-- A clear and concise description of what you would expect to happen. -->




Discussion:
I also have this issue (bump)
Pinging @richardliaw, @amogkam 
@maxzzze looks like a serialization error with the Trainer. We will take a look at this, but in the meantime can you downgrade your transformers version to 4.4. Also see https://github.com/ray-project/ray/issues/15439.
So it looks like this seems to work as soon as we disable the memory tracker:

```
trainer._memory_tracker = None
```

Will it be possible to expose an API to temporarily disable this? 

The other issue is https://github.com/huggingface/transformers/issues/11565, but we can resolve this there.

We should have tests that catch these regressions right?
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
I am having the same problem.
Disabling the memory tracker worked for me.
BUT, then I ran into #11565 as well

Resolving patch:
diff --git a/src/transformers/integrations.py b/src/transformers/integrations.py
--- a/src/transformers/integrations.py
+++ b/src/transformers/integrations.py
@@ -163,11 +163,21 @@ def _objective(trial, local_trainer, checkpoint_dir=None):
             local_trainer._tune_save_checkpoint()
             ray.tune.report(objective=local_trainer.objective, **metrics, done=True)
 
+    if not trainer._memory_tracker.skip_memory_metrics:
+        from .trainer_utils import TrainerMemoryTracker
+
+        logger.warning(
+            "Memory tracking for your Trainer is currently "
+            "enabled. Automatically disabling the memory tracker "
+            "since the memory tracker is not serializable."
+        )
+        trainer._memory_tracker = TrainerMemoryTracker(skip_memory_metrics=True)
+
     # The model and TensorBoard writer do not pickle so we have to remove them (if they exists)
     # while doing the ray hp search.
-
     _tb_writer = trainer.pop_callback(TensorBoardCallback)
     trainer.model = None
+
     # Setup default `resources_per_trial`.
     if "resources_per_trial" not in kwargs:
         # Default to 1 CPU and 1 GPU (if applicable) per trial.
@@ -194,7 +204,7 @@ def _objective(trial, local_trainer, checkpoint_dir=None):
         trainer.use_tune_checkpoints = True
         if kwargs["keep_checkpoints_num"] > 1:
             logger.warning(
-                f"Currently keeping {kwargs['keep_checkpoint_num']} checkpoints for each trial. "
+                f"Currently keeping {kwargs['keep_checkpoints_num']} checkpoints for each trial. "
                 "Checkpoints are usually huge, "
                 "consider setting `keep_checkpoints_num=1`."
             )


Test patch:
diff --git a/tests/test_trainer.py b/tests/test_trainer.py
--- a/tests/test_trainer.py
+++ b/tests/test_trainer.py
@@ -1287,7 +1287,7 @@ def setUp(self):
         self.n_epochs = args.num_train_epochs
         self.batch_size = args.train_batch_size
 
-    def test_hyperparameter_search(self):
+    def ray_hyperparameter_search(self):
         class MyTrialShortNamer(TrialShortNamer):
             DEFAULTS = {"a": 0, "b": 0}
 
@@ -1300,7 +1300,13 @@ def hp_space(trial):
             }
 
         def model_init(config):
-            model_config = RegressionModelConfig(a=config["a"], b=config["b"], double_output=False)
+            if config is None:
+                a = 0
+                b = 0
+            else:
+                a = config["a"]
+                b = config["b"]
+            model_config = RegressionModelConfig(a=a, b=b, double_output=False)
 
             return RegressionPreTrainedModel(model_config)
 
@@ -1323,3 +1329,14 @@ def hp_name(params):
             trainer.hyperparameter_search(
                 direction="minimize", hp_space=hp_space, hp_name=hp_name, backend="ray", n_trials=4
             )
+
+    def test_hyperparameter_search(self):
+        self.ray_hyperparameter_search()
+
+    def test_hyperparameter_search_ray_client(self):
+        import ray
+        from ray.util.client.ray_client_helpers import ray_start_client_server
+
+        with ray_start_client_server():
+            assert ray.util.client.ray.is_connected()
+            self.ray_hyperparameter_search()