AttributeError due to multi-processing using PyTorchBenchmark
Author: simonschoe
Pull request: no

Hi there,
likely my fault, but I can't find a proper solution yet. Tried to follow the structure as in `examples/benchmarking/run_benchmark.py`.

My Code:
```python
from transformers import AutoConfig, PyTorchBenchmark, PyTorchBenchmarkArguments


def main():
    
    config = AutoConfig.from_pretrained('roberta-base')

    # define args
    args = PyTorchBenchmarkArguments(
        models=['roberta-base'], 
        inference=False,
        training=True, 
        speed=True,
        memory=True,
        save_to_csv=True, 
        train_memory_csv_file=f'models/filmo-large/train_memory_benchmark.csv',
        train_time_csv_file=f'models/filmo-large/train_time_benchmark.csv',
        env_info_csv_file=f'models/filmo-large/env.csv',
        sequence_lengths=[64, 128, 256, 512],
        batch_sizes=[8, 16],
        fp16=True,
        multi_process=True,
    )

    # create benchmark
    benchmark = PyTorchBenchmark(
        configs=[config],
        args=args,
    )
    
    # run benchmark
    benchmark.run()


if __name__ == '__main__':
    main()
```

The error it yields:
```python
1 / 1
Traceback (most recent call last):
  File "c:/Users/.../lm-train-benchmark.py", line 47, in <module>
    main()
  File "c:/Users/.../lm-train-benchmark.py", line 43, in main
    benchmark.run()
  File "C:\Users\...\.venv\lib\site-packages\transformers\benchmark\benchmark_utils.py", line 715, in run
    memory, train_summary = self.train_memory(model_name, batch_size, sequence_length)
  File "C:\Users\...\.venv\lib\site-packages\transformers\benchmark\benchmark_utils.py", line 679, in train_memory
    return separate_process_wrapper_fn(self._train_memory, self.args.do_multi_processing)(*args, **kwargs)
  File "C:\Users\...\.venv\lib\site-packages\transformers\benchmark\benchmark_utils.py", line 101, in multi_process_func
    p.start()
  File "C:\Python\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Python\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Python\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
    reduction.dump(process_obj, to_child)
  File "C:\Python\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'separate_process_wrapper_fn.<locals>.multi_process_func.<locals>.wrapper_func'
PS C:\Users\...> Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Python\lib\multiprocessing\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input
```
Already tried to rearrange the order and place individual components on the top level, without success. I am grateful for any advice 😄 
Simon

Comments:
Same issue for me. 

When running
```python
python run_benchmark.py --no_speed --models a-ware/roberta-large-squad-classification --sequence_lengths 32 --batch_sizes 32
```
I get:
```
AttributeError: Can't pickle local object 'separate_process_wrapper_fn.<locals>.multi_process_func.<locals>.wrapper_func'
```

Anyone from the `transformers` team got a solution for this?

Can you try setting `multi_process=False`?

@patrickvonplaten If I remember correctly the error disappeared when setting `multi_process` to false. However, I figured I should set it to true in order to obtain performance estimates which are as close as possible to reality?

For PyTorch it's totally fine to set `multi_process=False` -> it's only in TF where the memory consumption is a bit off then

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

This issue is still present in the latest version (4.11.3). I ran the example benchmark shown [here](https://huggingface.co/transformers/benchmarks.html) in TensorFlow (2.6.0) and got the same error:

> AttributeError: Can't pickle local object 'separate_process_wrapper_fn.<locals>.multi_process_func.<locals>.wrapper_func