make test fails on `tests/test_benchmark_tf.py` 
Author: sergiovalmac
Pull request: no

## Environment inf
- `transformers` version: 4.12.0.dev0
- Platform: macOS-11.6-x86_64-i386-64bit
- Python version: 3.8.2
- PyTorch version (GPU?): 1.9.1 (False)
- Tensorflow version (GPU?): 2.6.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.3.5 (cpu)
- Jax version: 0.2.24
- JaxLib version: 0.1.73
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@Rocketknight1 

## Information
I cloned the repo and set up the environment installing `transformers` and `datasets`. Then, when I run the tests most of the tests in `tests/test_benchmark.py` fail.

The problem arises when using:
* [X ] the official example scripts: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Clean clone of `transformers` and `datasets`
2. Install all dependencies for both repos
3. Run tests
```
$ git clone https://github.com/huggingface/transformers
$ git clone https://github.com/huggingface/datasets
$ cd transformers
$ pip install -e ".[dev]"
$ cd ../datasets
$ pip install -e 
$ cd ../transformers
$ make test
```

This is the trace:
```
=================================== FAILURES ===================================
_________ TFBenchmarkTest.test_inference_encoder_decoder_with_configs __________
[gw3] darwin -- Python 3.8.2 /Users/sergiov/PycharmProjects/transformers/venv/bin/python

self = <tests.test_benchmark_tf.TFBenchmarkTest testMethod=test_inference_encoder_decoder_with_configs>

    def test_inference_encoder_decoder_with_configs(self):
        MODEL_ID = "patrickvonplaten/t5-tiny-random"
        config = AutoConfig.from_pretrained(MODEL_ID)
        benchmark_args = TensorFlowBenchmarkArguments(
            models=[MODEL_ID],
            training=False,
            inference=True,
            sequence_lengths=[8],
            batch_sizes=[1],
            multi_process=False,
        )
        benchmark = TensorFlowBenchmark(benchmark_args, configs=[config])
>       results = benchmark.run()

tests/test_benchmark_tf.py:161: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/transformers/benchmark/benchmark_utils.py:707: in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:676: in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark_tf.py:112: in _inference_memory
    return self._measure_memory(_inference)
src/transformers/benchmark/benchmark_tf.py:282: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-1' parent=10349 initial>
file = <_io.BytesIO object at 0x1c3958360>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
__________ BenchmarkTest.test_inference_encoder_decoder_with_configs ___________
[gw2] darwin -- Python 3.8.2 /Users/sergiov/PycharmProjects/transformers/venv/bin/python

self = <tests.test_benchmark.BenchmarkTest testMethod=test_inference_encoder_decoder_with_configs>

    def test_inference_encoder_decoder_with_configs(self):
        MODEL_ID = "sshleifer/tinier_bart"
        config = AutoConfig.from_pretrained(MODEL_ID)
        benchmark_args = PyTorchBenchmarkArguments(
            models=[MODEL_ID],
            training=False,
            inference=True,
            sequence_lengths=[8],
            batch_sizes=[1],
            multi_process=False,
        )
        benchmark = PyTorchBenchmark(benchmark_args, configs=[config])
>       results = benchmark.run()

tests/test_benchmark.py:178: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/transformers/benchmark/benchmark_utils.py:707: in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:676: in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark.py:69: in _inference_memory
    return self._measure_memory(_inference)
src/transformers/benchmark/benchmark.py:258: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-1' parent=10347 initial>
file = <_io.BytesIO object at 0x1c312c540>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
___________________ BenchmarkTest.test_inference_no_configs ____________________
[gw2] darwin -- Python 3.8.2 /Users/sergiov/PycharmProjects/transformers/venv/bin/python

self = <tests.test_benchmark.BenchmarkTest testMethod=test_inference_no_configs>

    def test_inference_no_configs(self):
        MODEL_ID = "sshleifer/tiny-gpt2"
        benchmark_args = PyTorchBenchmarkArguments(
            models=[MODEL_ID],
            training=False,
            inference=True,
            sequence_lengths=[8],
            batch_sizes=[1],
            multi_process=False,
        )
        benchmark = PyTorchBenchmark(benchmark_args)
>       results = benchmark.run()

tests/test_benchmark.py:47: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/transformers/benchmark/benchmark_utils.py:707: in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:676: in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark.py:69: in _inference_memory
    return self._measure_memory(_inference)
src/transformers/benchmark/benchmark.py:258: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-2' parent=10347 initial>
file = <_io.BytesIO object at 0x1c30a6770>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
_______________ TFBenchmarkTest.test_inference_no_configs_eager ________________
[gw3] darwin -- Python 3.8.2 /Users/sergiov/PycharmProjects/transformers/venv/bin/python

self = <tests.test_benchmark_tf.TFBenchmarkTest testMethod=test_inference_no_configs_eager>

    def test_inference_no_configs_eager(self):
        MODEL_ID = "sshleifer/tiny-gpt2"
        benchmark_args = TensorFlowBenchmarkArguments(
            models=[MODEL_ID],
            training=False,
            inference=True,
            sequence_lengths=[8],
            batch_sizes=[1],
            eager_mode=True,
            multi_process=False,
        )
        benchmark = TensorFlowBenchmark(benchmark_args)
>       results = benchmark.run()

tests/test_benchmark_tf.py:50: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/transformers/benchmark/benchmark_utils.py:707: in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:676: in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark_tf.py:112: in _inference_memory
    return self._measure_memory(_inference)
src/transformers/benchmark/benchmark_tf.py:282: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-2' parent=10349 initial>
file = <_io.BytesIO object at 0x1c3a92450>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
____________ BenchmarkTest.test_inference_no_configs_only_pretrain _____________
[gw2] darwin -- Python 3.8.2 /Users/sergiov/PycharmProjects/transformers/venv/bin/python

self = <tests.test_benchmark.BenchmarkTest testMethod=test_inference_no_configs_only_pretrain>

    def test_inference_no_configs_only_pretrain(self):
        MODEL_ID = "sgugger/tiny-distilbert-classification"
        benchmark_args = PyTorchBenchmarkArguments(
            models=[MODEL_ID],
            training=False,
            inference=True,
            sequence_lengths=[8],
            batch_sizes=[1],
            multi_process=False,
            only_pretrain_model=True,
        )
        benchmark = PyTorchBenchmark(benchmark_args)
>       results = benchmark.run()

tests/test_benchmark.py:63: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/transformers/benchmark/benchmark_utils.py:707: in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:676: in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark.py:69: in _inference_memory
    return self._measure_memory(_inference)
src/transformers/benchmark/benchmark.py:258: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-3' parent=10347 initial>
file = <_io.BytesIO object at 0x1c30a6220>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
_______________ TFBenchmarkTest.test_inference_no_configs_graph ________________
[gw3] darwin -- Python 3.8.2 /Users/sergiov/PycharmProjects/transformers/venv/bin/python

self = <tests.test_benchmark_tf.TFBenchmarkTest testMethod=test_inference_no_configs_graph>

    def test_inference_no_configs_graph(self):
        MODEL_ID = "sshleifer/tiny-gpt2"
        benchmark_args = TensorFlowBenchmarkArguments(
            models=[MODEL_ID],
            training=False,
            inference=True,
            sequence_lengths=[8],
            batch_sizes=[1],
            multi_process=False,
        )
        benchmark = TensorFlowBenchmark(benchmark_args)
>       results = benchmark.run()

tests/test_benchmark_tf.py:81: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/transformers/benchmark/benchmark_utils.py:707: in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:676: in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark_tf.py:112: in _inference_memory
    return self._measure_memory(_inference)
src/transformers/benchmark/benchmark_tf.py:282: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-3' parent=10349 initial>
file = <_io.BytesIO object at 0x1c3aaa040>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
____________ BenchmarkTest.test_inference_no_model_no_architectures ____________
[gw2] darwin -- Python 3.8.2 /Users/sergiov/PycharmProjects/transformers/venv/bin/python

self = <tests.test_benchmark.BenchmarkTest testMethod=test_inference_no_model_no_architectures>

    def test_inference_no_model_no_architectures(self):
        MODEL_ID = "sshleifer/tiny-gpt2"
        config = AutoConfig.from_pretrained(MODEL_ID)
        # set architectures equal to `None`
        config.architectures = None
        benchmark_args = PyTorchBenchmarkArguments(
            models=[MODEL_ID],
            training=True,
            inference=True,
            sequence_lengths=[8],
            batch_sizes=[1],
            multi_process=False,
        )
        benchmark = PyTorchBenchmark(benchmark_args, configs=[config])
>       results = benchmark.run()

tests/test_benchmark.py:114: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/transformers/benchmark/benchmark_utils.py:707: in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:676: in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark.py:69: in _inference_memory
    return self._measure_memory(_inference)
src/transformers/benchmark/benchmark.py:258: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-4' parent=10347 initial>
file = <_io.BytesIO object at 0x1c32df090>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
___________ TFBenchmarkTest.test_inference_no_configs_only_pretrain ____________
[gw3] darwin -- Python 3.8.2 /Users/sergiov/PycharmProjects/transformers/venv/bin/python

self = <tests.test_benchmark_tf.TFBenchmarkTest testMethod=test_inference_no_configs_only_pretrain>

    def test_inference_no_configs_only_pretrain(self):
        MODEL_ID = "sgugger/tiny-distilbert-classification"
        benchmark_args = TensorFlowBenchmarkArguments(
            models=[MODEL_ID],
            training=False,
            inference=True,
            sequence_lengths=[8],
            batch_sizes=[1],
            multi_process=False,
            only_pretrain_model=True,
        )
        benchmark = TensorFlowBenchmark(benchmark_args)
>       results = benchmark.run()

tests/test_benchmark_tf.py:66: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/transformers/benchmark/benchmark_utils.py:707: in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:676: in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark_tf.py:112: in _inference_memory
    return self._measure_memory(_inference)
src/transformers/benchmark/benchmark_tf.py:282: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-4' parent=10349 initial>
file = <_io.BytesIO object at 0x1c3af8c70>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
______________ TFBenchmarkTest.test_inference_with_configs_eager _______________
[gw3] darwin -- Python 3.8.2 /Users/sergiov/PycharmProjects/transformers/venv/bin/python

self = <tests.test_benchmark_tf.TFBenchmarkTest testMethod=test_inference_with_configs_eager>

    def test_inference_with_configs_eager(self):
        MODEL_ID = "sshleifer/tiny-gpt2"
        config = AutoConfig.from_pretrained(MODEL_ID)
        benchmark_args = TensorFlowBenchmarkArguments(
            models=[MODEL_ID],
            training=False,
            inference=True,
            sequence_lengths=[8],
            batch_sizes=[1],
            eager_mode=True,
            multi_process=False,
        )
        benchmark = TensorFlowBenchmark(benchmark_args, [config])
>       results = benchmark.run()

tests/test_benchmark_tf.py:98: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/transformers/benchmark/benchmark_utils.py:707: in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:676: in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark_tf.py:112: in _inference_memory
    return self._measure_memory(_inference)
src/transformers/benchmark/benchmark_tf.py:282: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-5' parent=10349 initial>
file = <_io.BytesIO object at 0x1c39cca90>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
___________________ BenchmarkTest.test_inference_torchscript ___________________
[gw2] darwin -- Python 3.8.2 /Users/sergiov/PycharmProjects/transformers/venv/bin/python

self = <tests.test_benchmark.BenchmarkTest testMethod=test_inference_torchscript>

    def test_inference_torchscript(self):
        MODEL_ID = "sshleifer/tiny-gpt2"
        benchmark_args = PyTorchBenchmarkArguments(
            models=[MODEL_ID],
            training=False,
            inference=True,
            torchscript=True,
            sequence_lengths=[8],
            batch_sizes=[1],
            multi_process=False,
        )
        benchmark = PyTorchBenchmark(benchmark_args)
>       results = benchmark.run()

tests/test_benchmark.py:79: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/transformers/benchmark/benchmark_utils.py:707: in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:676: in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark.py:69: in _inference_memory
    return self._measure_memory(_inference)
src/transformers/benchmark/benchmark.py:258: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-5' parent=10347 initial>
file = <_io.BytesIO object at 0x1c34189a0>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
______________ TFBenchmarkTest.test_inference_with_configs_graph _______________
[gw3] darwin -- Python 3.8.2 /Users/sergiov/PycharmProjects/transformers/venv/bin/python

self = <tests.test_benchmark_tf.TFBenchmarkTest testMethod=test_inference_with_configs_graph>

    def test_inference_with_configs_graph(self):
        MODEL_ID = "sshleifer/tiny-gpt2"
        config = AutoConfig.from_pretrained(MODEL_ID)
        benchmark_args = TensorFlowBenchmarkArguments(
            models=[MODEL_ID],
            training=False,
            inference=True,
            sequence_lengths=[8],
            batch_sizes=[1],
            multi_process=False,
        )
        benchmark = TensorFlowBenchmark(benchmark_args, [config])
>       results = benchmark.run()

tests/test_benchmark_tf.py:114: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/transformers/benchmark/benchmark_utils.py:707: in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:676: in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark_tf.py:112: in _inference_memory
    return self._measure_memory(_inference)
src/transformers/benchmark/benchmark_tf.py:282: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-6' parent=10349 initial>
file = <_io.BytesIO object at 0x1c38ba860>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
__________________ BenchmarkTest.test_inference_with_configs ___________________
[gw2] darwin -- Python 3.8.2 /Users/sergiov/PycharmProjects/transformers/venv/bin/python

self = <tests.test_benchmark.BenchmarkTest testMethod=test_inference_with_configs>

    def test_inference_with_configs(self):
        MODEL_ID = "sshleifer/tiny-gpt2"
        config = AutoConfig.from_pretrained(MODEL_ID)
        benchmark_args = PyTorchBenchmarkArguments(
            models=[MODEL_ID],
            training=False,
            inference=True,
            sequence_lengths=[8],
            batch_sizes=[1],
            multi_process=False,
        )
        benchmark = PyTorchBenchmark(benchmark_args, configs=[config])
>       results = benchmark.run()

tests/test_benchmark.py:162: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/transformers/benchmark/benchmark_utils.py:707: in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:676: in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark.py:69: in _inference_memory
    return self._measure_memory(_inference)
src/transformers/benchmark/benchmark.py:258: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-6' parent=10347 initial>
file = <_io.BytesIO object at 0x1c2f605e0>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
_____________________ TFBenchmarkTest.test_save_csv_files ______________________
[gw3] darwin -- Python 3.8.2 /Users/sergiov/PycharmProjects/transformers/venv/bin/python

self = <tests.test_benchmark_tf.TFBenchmarkTest testMethod=test_save_csv_files>

    def test_save_csv_files(self):
        MODEL_ID = "sshleifer/tiny-gpt2"
        with tempfile.TemporaryDirectory() as tmp_dir:
            benchmark_args = TensorFlowBenchmarkArguments(
                models=[MODEL_ID],
                inference=True,
                save_to_csv=True,
                sequence_lengths=[8],
                batch_sizes=[1],
                inference_time_csv_file=os.path.join(tmp_dir, "inf_time.csv"),
                inference_memory_csv_file=os.path.join(tmp_dir, "inf_mem.csv"),
                env_info_csv_file=os.path.join(tmp_dir, "env.csv"),
                multi_process=False,
            )
            benchmark = TensorFlowBenchmark(benchmark_args)
>           benchmark.run()

tests/test_benchmark_tf.py:197: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/transformers/benchmark/benchmark_utils.py:707: in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:676: in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark_tf.py:112: in _inference_memory
    return self._measure_memory(_inference)
src/transformers/benchmark/benchmark_tf.py:282: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-7' parent=10349 initial>
file = <_io.BytesIO object at 0x1bb6e7220>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
______________________ BenchmarkTest.test_save_csv_files _______________________
[gw2] darwin -- Python 3.8.2 /Users/sergiov/PycharmProjects/transformers/venv/bin/python

self = <tests.test_benchmark.BenchmarkTest testMethod=test_save_csv_files>

    def test_save_csv_files(self):
        MODEL_ID = "sshleifer/tiny-gpt2"
        with tempfile.TemporaryDirectory() as tmp_dir:
            benchmark_args = PyTorchBenchmarkArguments(
                models=[MODEL_ID],
                training=True,
                inference=True,
                save_to_csv=True,
                sequence_lengths=[8],
                batch_sizes=[1],
                inference_time_csv_file=os.path.join(tmp_dir, "inf_time.csv"),
                train_memory_csv_file=os.path.join(tmp_dir, "train_mem.csv"),
                inference_memory_csv_file=os.path.join(tmp_dir, "inf_mem.csv"),
                train_time_csv_file=os.path.join(tmp_dir, "train_time.csv"),
                env_info_csv_file=os.path.join(tmp_dir, "env.csv"),
                multi_process=False,
            )
            benchmark = PyTorchBenchmark(benchmark_args)
>           benchmark.run()

tests/test_benchmark.py:232: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/transformers/benchmark/benchmark_utils.py:707: in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:676: in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark.py:69: in _inference_memory
    return self._measure_memory(_inference)
src/transformers/benchmark/benchmark.py:258: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-7' parent=10347 initial>
file = <_io.BytesIO object at 0x1c30a64f0>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
_______________________ BenchmarkTest.test_trace_memory ________________________
[gw2] darwin -- Python 3.8.2 /Users/sergiov/PycharmProjects/transformers/venv/bin/python

self = <tests.test_benchmark.BenchmarkTest testMethod=test_trace_memory>

    def test_trace_memory(self):
        MODEL_ID = "sshleifer/tiny-gpt2"
    
        def _check_summary_is_not_empty(summary):
            self.assertTrue(hasattr(summary, "sequential"))
            self.assertTrue(hasattr(summary, "cumulative"))
            self.assertTrue(hasattr(summary, "current"))
            self.assertTrue(hasattr(summary, "total"))
    
        with tempfile.TemporaryDirectory() as tmp_dir:
            benchmark_args = PyTorchBenchmarkArguments(
                models=[MODEL_ID],
                training=True,
                inference=True,
                sequence_lengths=[8],
                batch_sizes=[1],
                log_filename=os.path.join(tmp_dir, "log.txt"),
                log_print=True,
                trace_memory_line_by_line=True,
                multi_process=False,
            )
            benchmark = PyTorchBenchmark(benchmark_args)
>           result = benchmark.run()

tests/test_benchmark.py:261: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/transformers/benchmark/benchmark_utils.py:707: in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:676: in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark.py:69: in _inference_memory
    return self._measure_memory(_inference)
src/transformers/benchmark/benchmark.py:258: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-8' parent=10347 initial>
file = <_io.BytesIO object at 0x1c32298b0>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
____________________ TFBenchmarkTest.test_train_no_configs _____________________
[gw3] darwin -- Python 3.8.2 /Users/sergiov/PycharmProjects/transformers/venv/bin/python

self = <tests.test_benchmark_tf.TFBenchmarkTest testMethod=test_train_no_configs>

    def test_train_no_configs(self):
        MODEL_ID = "sshleifer/tiny-gpt2"
        benchmark_args = TensorFlowBenchmarkArguments(
            models=[MODEL_ID],
            training=True,
            inference=False,
            sequence_lengths=[8],
            batch_sizes=[1],
            multi_process=False,
        )
        benchmark = TensorFlowBenchmark(benchmark_args)
>       results = benchmark.run()

tests/test_benchmark_tf.py:129: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/transformers/benchmark/benchmark_utils.py:715: in run
    memory, train_summary = self.train_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:679: in train_memory
    return separate_process_wrapper_fn(self._train_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark_tf.py:123: in _train_memory
    return self._measure_memory(_train)
src/transformers/benchmark/benchmark_tf.py:282: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-8' parent=10349 initial>
file = <_io.BytesIO object at 0x1c39278b0>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
____________ BenchmarkTest.test_train_encoder_decoder_with_configs _____________
[gw2] darwin -- Python 3.8.2 /Users/sergiov/PycharmProjects/transformers/venv/bin/python

self = <tests.test_benchmark.BenchmarkTest testMethod=test_train_encoder_decoder_with_configs>

    def test_train_encoder_decoder_with_configs(self):
        MODEL_ID = "sshleifer/tinier_bart"
        config = AutoConfig.from_pretrained(MODEL_ID)
        benchmark_args = PyTorchBenchmarkArguments(
            models=[MODEL_ID],
            training=True,
            inference=True,
            sequence_lengths=[8],
            batch_sizes=[1],
            multi_process=False,
        )
        benchmark = PyTorchBenchmark(benchmark_args, configs=[config])
>       results = benchmark.run()

tests/test_benchmark.py:210: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/transformers/benchmark/benchmark_utils.py:707: in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:676: in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark.py:69: in _inference_memory
    return self._measure_memory(_inference)
src/transformers/benchmark/benchmark.py:258: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-9' parent=10347 initial>
file = <_io.BytesIO object at 0x1c3745360>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
___________________ TFBenchmarkTest.test_train_with_configs ____________________
[gw3] darwin -- Python 3.8.2 /Users/sergiov/PycharmProjects/transformers/venv/bin/python

self = <tests.test_benchmark_tf.TFBenchmarkTest testMethod=test_train_with_configs>

    def test_train_with_configs(self):
        MODEL_ID = "sshleifer/tiny-gpt2"
        config = AutoConfig.from_pretrained(MODEL_ID)
        benchmark_args = TensorFlowBenchmarkArguments(
            models=[MODEL_ID],
            training=True,
            inference=False,
            sequence_lengths=[8],
            batch_sizes=[1],
            multi_process=False,
        )
        benchmark = TensorFlowBenchmark(benchmark_args, [config])
>       results = benchmark.run()

tests/test_benchmark_tf.py:145: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/transformers/benchmark/benchmark_utils.py:715: in run
    memory, train_summary = self.train_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:679: in train_memory
    return separate_process_wrapper_fn(self._train_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark_tf.py:123: in _train_memory
    return self._measure_memory(_train)
src/transformers/benchmark/benchmark_tf.py:282: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-9' parent=10349 initial>
file = <_io.BytesIO object at 0x1c3c35ae0>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
_____________________ BenchmarkTest.test_train_no_configs ______________________
[gw2] darwin -- Python 3.8.2 /Users/sergiov/PycharmProjects/transformers/venv/bin/python

self = <tests.test_benchmark.BenchmarkTest testMethod=test_train_no_configs>

    def test_train_no_configs(self):
        MODEL_ID = "sshleifer/tiny-gpt2"
        benchmark_args = PyTorchBenchmarkArguments(
            models=[MODEL_ID],
            training=True,
            inference=False,
            sequence_lengths=[8],
            batch_sizes=[1],
            multi_process=False,
        )
        benchmark = PyTorchBenchmark(benchmark_args)
>       results = benchmark.run()

tests/test_benchmark.py:129: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/transformers/benchmark/benchmark_utils.py:715: in run
    memory, train_summary = self.train_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:679: in train_memory
    return separate_process_wrapper_fn(self._train_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark.py:79: in _train_memory
    return self._measure_memory(_train)
src/transformers/benchmark/benchmark.py:258: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-10' parent=10347 initial>
file = <_io.BytesIO object at 0x1c379a180>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
____________________ BenchmarkTest.test_train_with_configs _____________________
[gw2] darwin -- Python 3.8.2 /Users/sergiov/PycharmProjects/transformers/venv/bin/python

self = <tests.test_benchmark.BenchmarkTest testMethod=test_train_with_configs>

    def test_train_with_configs(self):
        MODEL_ID = "sshleifer/tiny-gpt2"
        config = AutoConfig.from_pretrained(MODEL_ID)
        benchmark_args = PyTorchBenchmarkArguments(
            models=[MODEL_ID],
            training=True,
            inference=False,
            sequence_lengths=[8],
            batch_sizes=[1],
            multi_process=False,
        )
        benchmark = PyTorchBenchmark(benchmark_args, configs=[config])
>       results = benchmark.run()

tests/test_benchmark.py:194: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/transformers/benchmark/benchmark_utils.py:715: in run
    memory, train_summary = self.train_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:679: in train_memory
    return separate_process_wrapper_fn(self._train_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark.py:79: in _train_memory
    return self._measure_memory(_train)
src/transformers/benchmark/benchmark.py:258: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-11' parent=10347 initial>
file = <_io.BytesIO object at 0x1cb8ae2c0>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
=============================== warnings summary ===============================
venv/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:22
venv/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:22
venv/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:22
venv/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:22
venv/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:22
venv/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:22
venv/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:22
venv/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:22
venv/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:22
  /Users/sergiov/PycharmProjects/transformers/venv/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

tests/test_feature_extraction_wav2vec2.py: 7 warnings
tests/test_feature_extraction_speech_to_text.py: 7 warnings
  /Users/sergiov/PycharmProjects/transformers/venv/lib/python3.8/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
    return array(a, dtype, copy=False, order=order)

tests/test_generation_beam_search.py::BeamSearchTest::test_beam_scorer_update
tests/test_modeling_bart.py::BartHeadTests::test_generate_beam_search
tests/test_modeling_bert.py::BertModelTest::test_beam_sample_generate
tests/test_modeling_bert_generation.py::BertGenerationEncoderTest::test_beam_sample_generate
tests/test_modeling_big_bird.py::BigBirdModelTest::test_auto_padding
tests/test_modeling_blenderbot.py::BlenderbotModelTest::test_beam_sample_generate
tests/test_modeling_ctrl.py::CTRLModelTest::test_attention_outputs
  /Users/sergiov/PycharmProjects/transformers/venv/lib/python3.8/site-packages/torch/_tensor.py:575: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
  To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at  ../aten/src/ATen/native/BinaryOps.cpp:467.)
    return torch.floor_divide(self, other)

tests/test_generation_flax_logits_process.py: 36 warnings
tests/test_modeling_flax_bart.py: 46 warnings
tests/test_modeling_flax_big_bird.py: 5 warnings
  /Users/sergiov/PycharmProjects/transformers/venv/lib/python3.8/site-packages/jax/_src/ops/scatter.py:382: DeprecationWarning: index_update is deprecated. Use x.at[idx].set(y) instead.
    warnings.warn("index_update is deprecated. Use x.at[idx].set(y) instead.",

tests/test_generation_stopping_criteria.py::StoppingCriteriaTestCase::test_max_new_tokens_criteria
  /Users/sergiov/PycharmProjects/transformers/src/transformers/generation_stopping_criteria.py:74: FutureWarning: The class `MaxNewTokensCriteria` is deprecated. Please use `MaxLengthCriteria(max_length=10)` with `max_length = start_length + max_new_tokens` instead.
    warnings.warn(

tests/test_model_card.py::ModelCardTester::test_model_card_common_properties
tests/test_model_card.py::ModelCardTester::test_model_card_from_and_save_pretrained
tests/test_model_card.py::ModelCardTester::test_model_card_to_json_file
tests/test_model_card.py::ModelCardTester::test_model_card_to_json_string
  /Users/sergiov/PycharmProjects/transformers/src/transformers/modelcard.py:92: FutureWarning: The class `ModelCard` is deprecated and will be removed in version 5 of Transformers
    warnings.warn(

tests/test_model_card.py::ModelCardTester::test_model_card_from_and_save_pretrained
  /Users/sergiov/PycharmProjects/transformers/src/transformers/models/auto/configuration_auto.py:343: FutureWarning: ALL_PRETRAINED_CONFIG_ARCHIVE_MAP is deprecated and will be removed in v5 of Transformers. It does not contain all available model checkpoints, far from it. Checkout hf.co/models for that.
    warnings.warn(

tests/test_modeling_auto.py::AutoModelTest::test_from_identifier_from_model_type
tests/test_modeling_auto.py::AutoModelTest::test_from_pretrained_identifier
  /Users/sergiov/PycharmProjects/transformers/src/transformers/models/auto/modeling_auto.py:684: FutureWarning: The class `AutoModelWithLMHead` is deprecated and will be removed in a future version. Please use `AutoModelForCausalLM` for causal language models, `AutoModelForMaskedLM` for masked language models and `AutoModelForSeq2SeqLM` for encoder-decoder models.
    warnings.warn(

tests/test_modeling_auto.py::AutoModelTest::test_from_pretrained_dynamic_model
  /Users/sergiov/PycharmProjects/transformers/src/transformers/models/auto/auto_factory.py:407: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
    logger.warn(

tests/test_modeling_bart.py: 4 warnings
tests/test_modeling_bert.py: 2 warnings
tests/test_modeling_bert_generation.py: 2 warnings
tests/test_modeling_bigbird_pegasus.py: 4 warnings
tests/test_modeling_blenderbot_small.py: 4 warnings
tests/test_modeling_blenderbot.py: 4 warnings
tests/test_modeling_ctrl.py: 2 warnings
  /Users/sergiov/PycharmProjects/transformers/src/transformers/generation_utils.py:2036: UserWarning: `max_length` is deprecated in this function, use `stopping_criteria=StoppingCriteriaList(MaxLengthCriteria(max_length=max_length))` instead.
    warnings.warn(

tests/test_modeling_bart.py: 6 warnings
tests/test_modeling_bert.py: 3 warnings
tests/test_modeling_bert_generation.py: 3 warnings
tests/test_modeling_bigbird_pegasus.py: 6 warnings
tests/test_modeling_blenderbot_small.py: 6 warnings
tests/test_modeling_blenderbot.py: 6 warnings
tests/test_modeling_ctrl.py: 3 warnings
  /Users/sergiov/PycharmProjects/transformers/src/transformers/generation_utils.py:1731: UserWarning: `max_length` is deprecated in this function, use `stopping_criteria=StoppingCriteriaList(MaxLengthCriteria(max_length=max_length))` instead.
    warnings.warn(

tests/test_benchmark.py::BenchmarkTest::test_inference_torchscript
  /Users/sergiov/PycharmProjects/transformers/src/transformers/models/gpt2/modeling_gpt2.py:196: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
    attn_weights = attn_weights / (float(value.size(-1)) ** 0.5)

tests/test_modeling_bart.py: 6 warnings
tests/test_modeling_bert_generation.py: 3 warnings
tests/test_modeling_bert.py: 3 warnings
tests/test_modeling_bigbird_pegasus.py: 6 warnings
tests/test_modeling_blenderbot_small.py: 6 warnings
tests/test_modeling_blenderbot.py: 6 warnings
tests/test_modeling_ctrl.py: 3 warnings
  /Users/sergiov/PycharmProjects/transformers/src/transformers/generation_utils.py:1242: UserWarning: `max_length` is deprecated in this function, use `stopping_criteria=StoppingCriteriaList(MaxLengthCriteria(max_length=max_length))` instead.
    warnings.warn(

tests/test_modeling_bart.py: 4 warnings
tests/test_modeling_bert_generation.py: 2 warnings
tests/test_modeling_bert.py: 2 warnings
tests/test_modeling_bigbird_pegasus.py: 4 warnings
tests/test_modeling_blenderbot_small.py: 4 warnings
tests/test_modeling_blenderbot.py: 4 warnings
tests/test_modeling_ctrl.py: 2 warnings
  /Users/sergiov/PycharmProjects/transformers/src/transformers/generation_beam_search.py:196: UserWarning: Passing `max_length` to BeamSearchScorer is deprecated and has no effect. `max_length` should be passed directly to `beam_search(...)`, `beam_sample(...)`, or `group_beam_search(...)`.
    warnings.warn(

tests/test_modeling_bart.py: 4 warnings
tests/test_modeling_bert_generation.py: 2 warnings
tests/test_modeling_bert.py: 2 warnings
tests/test_modeling_bigbird_pegasus.py: 4 warnings
tests/test_modeling_blenderbot_small.py: 4 warnings
tests/test_modeling_blenderbot.py: 4 warnings
tests/test_modeling_ctrl.py: 2 warnings
  /Users/sergiov/PycharmProjects/transformers/src/transformers/generation_utils.py:2331: UserWarning: `max_length` is deprecated in this function, use `stopping_criteria=StoppingCriteriaList(MaxLengthCriteria(max_length=max_length))` instead.
    warnings.warn(

tests/test_modeling_bert_generation.py: 2 warnings
tests/test_modeling_bart.py: 4 warnings
tests/test_modeling_bert.py: 2 warnings
tests/test_modeling_blenderbot_small.py: 4 warnings
tests/test_modeling_bigbird_pegasus.py: 4 warnings
tests/test_modeling_ctrl.py: 2 warnings
tests/test_modeling_blenderbot.py: 4 warnings
  /Users/sergiov/PycharmProjects/transformers/src/transformers/generation_utils.py:1479: UserWarning: `max_length` is deprecated in this function, use `stopping_criteria=StoppingCriteriaList(MaxLengthCriteria(max_length=max_length))` instead.
    warnings.warn(

tests/test_modeling_albert.py: 3 warnings
tests/test_modeling_bert.py: 3 warnings
tests/test_modeling_distilbert.py: 3 warnings
tests/test_modeling_electra.py: 3 warnings
  /Users/sergiov/PycharmProjects/transformers/venv/lib/python3.8/site-packages/torch/fx/symbolic_trace.py:434: UserWarning: Was not able to add assertion to guarantee correct inputs to specialized function. It is up to the user to make sure that your inputs match the inputs you specialized the function with.
    torch.warnings.warn(

tests/test_modeling_common.py::ModelUtilsTest::test_model_from_pretrained_with_different_pretrained_model_name
  /Users/sergiov/PycharmProjects/transformers/src/transformers/configuration_utils.py:495: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
    logger.warn(

tests/test_modeling_flaubert.py::FlaubertModelTest::test_flaubert_lm_head
tests/test_modeling_flaubert.py::FlaubertModelTest::test_model_outputs_equivalence
tests/test_modeling_flaubert.py::FlaubertModelTest::test_training
  /Users/sergiov/PycharmProjects/transformers/venv/lib/python3.8/site-packages/torch/nn/_reduction.py:13: UserWarning: reduction='elementwise_mean' is deprecated, please use reduction='mean' instead.
    warnings.warn("reduction='elementwise_mean' is deprecated, please use reduction='mean' instead.")

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ============================
FAILED tests/test_benchmark_tf.py::TFBenchmarkTest::test_inference_encoder_decoder_with_configs
FAILED tests/test_benchmark.py::BenchmarkTest::test_inference_encoder_decoder_with_configs
FAILED tests/test_benchmark.py::BenchmarkTest::test_inference_no_configs - At...
FAILED tests/test_benchmark_tf.py::TFBenchmarkTest::test_inference_no_configs_eager
FAILED tests/test_benchmark.py::BenchmarkTest::test_inference_no_configs_only_pretrain
FAILED tests/test_benchmark_tf.py::TFBenchmarkTest::test_inference_no_configs_graph
FAILED tests/test_benchmark.py::BenchmarkTest::test_inference_no_model_no_architectures
FAILED tests/test_benchmark_tf.py::TFBenchmarkTest::test_inference_no_configs_only_pretrain
FAILED tests/test_benchmark_tf.py::TFBenchmarkTest::test_inference_with_configs_eager
FAILED tests/test_benchmark.py::BenchmarkTest::test_inference_torchscript - A...
FAILED tests/test_benchmark_tf.py::TFBenchmarkTest::test_inference_with_configs_graph
FAILED tests/test_benchmark.py::BenchmarkTest::test_inference_with_configs - ...
FAILED tests/test_benchmark_tf.py::TFBenchmarkTest::test_save_csv_files - Att...
FAILED tests/test_benchmark.py::BenchmarkTest::test_save_csv_files - Attribut...
FAILED tests/test_benchmark.py::BenchmarkTest::test_trace_memory - AttributeE...
FAILED tests/test_benchmark_tf.py::TFBenchmarkTest::test_train_no_configs - A...
FAILED tests/test_benchmark.py::BenchmarkTest::test_train_encoder_decoder_with_configs
FAILED tests/test_benchmark_tf.py::TFBenchmarkTest::test_train_with_configs
FAILED tests/test_benchmark.py::BenchmarkTest::test_train_no_configs - Attrib...
FAILED tests/test_benchmark.py::BenchmarkTest::test_train_with_configs - Attr...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/threading.py:306: KeyboardInterrupt
(to show a full traceback on KeyboardInterrupt use --full-trace)
==== 20 failed, 1520 passed, 335 skipped, 297 warnings in 214.63s (0:03:34) ====
```

## Expected behavior
I expect all tests to pass


Comments:
This seems similar to the second part of this other [issue 13227](https://github.com/huggingface/transformers/issues/13227).

Hi, I'm testing this locally. I also get some errors in `make test` - I'll ask the team about that. However, you can also check individual modules e.g. with `pytest test_benchmark_tf.py`. When I do this, the file passes. Can you confirm that the same thing happens for you?

Hi, thanks for looking into this! 
No, I still have errors when run locally:

```
% pytest tests/test_benchmark.py
==================================================================================================================== test session starts =====================================================================================================================
platform darwin -- Python 3.8.2, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /Users/sergiov/PycharmProjects/transformers, configfile: setup.cfg
plugins: xdist-2.4.0, timeout-2.0.1, dash-2.0.0, forked-1.3.0
collected 13 items

tests/test_benchmark.py FsFFFFFFFFFsF                                                                                                                                                                                                                  [100%]

========================================================================================================================== FAILURES ==========================================================================================================================
_________________________________________________________________________________________________ BenchmarkTest.test_inference_encoder_decoder_with_configs __________________________________________________________________________________________________

self = <tests.test_benchmark.BenchmarkTest testMethod=test_inference_encoder_decoder_with_configs>

    def test_inference_encoder_decoder_with_configs(self):
        MODEL_ID = "sshleifer/tinier_bart"
        config = AutoConfig.from_pretrained(MODEL_ID)
        benchmark_args = PyTorchBenchmarkArguments(
            models=[MODEL_ID],
            training=False,
            inference=True,
            sequence_lengths=[8],
            batch_sizes=[1],
            multi_process=False,
        )
        benchmark = PyTorchBenchmark(benchmark_args, configs=[config])
>       results = benchmark.run()

tests/test_benchmark.py:178:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/transformers/benchmark/benchmark_utils.py:707: in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:676: in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark.py:69: in _inference_memory
    return self._measure_memory(_inference)
src/transformers/benchmark/benchmark.py:258: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-1' parent=1591 initial>, file = <_io.BytesIO object at 0x19b6ef680>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
-------------------------------------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------------------------------------
1 / 1
__________________________________________________________________________________________________________ BenchmarkTest.test_inference_no_configs ___________________________________________________________________________________________________________

self = <tests.test_benchmark.BenchmarkTest testMethod=test_inference_no_configs>

    def test_inference_no_configs(self):
        MODEL_ID = "sshleifer/tiny-gpt2"
        benchmark_args = PyTorchBenchmarkArguments(
            models=[MODEL_ID],
            training=False,
            inference=True,
            sequence_lengths=[8],
            batch_sizes=[1],
            multi_process=False,
        )
        benchmark = PyTorchBenchmark(benchmark_args)
>       results = benchmark.run()

tests/test_benchmark.py:47:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/transformers/benchmark/benchmark_utils.py:707: in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:676: in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark.py:69: in _inference_memory
    return self._measure_memory(_inference)
src/transformers/benchmark/benchmark.py:258: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-2' parent=1591 initial>, file = <_io.BytesIO object at 0x1a3957310>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
-------------------------------------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------------------------------------
1 / 1
___________________________________________________________________________________________________ BenchmarkTest.test_inference_no_configs_only_pretrain ____________________________________________________________________________________________________

self = <tests.test_benchmark.BenchmarkTest testMethod=test_inference_no_configs_only_pretrain>

    def test_inference_no_configs_only_pretrain(self):
        MODEL_ID = "sgugger/tiny-distilbert-classification"
        benchmark_args = PyTorchBenchmarkArguments(
            models=[MODEL_ID],
            training=False,
            inference=True,
            sequence_lengths=[8],
            batch_sizes=[1],
            multi_process=False,
            only_pretrain_model=True,
        )
        benchmark = PyTorchBenchmark(benchmark_args)
>       results = benchmark.run()

tests/test_benchmark.py:63:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/transformers/benchmark/benchmark_utils.py:707: in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:676: in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark.py:69: in _inference_memory
    return self._measure_memory(_inference)
src/transformers/benchmark/benchmark.py:258: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-3' parent=1591 initial>, file = <_io.BytesIO object at 0x1a38e63b0>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
-------------------------------------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------------------------------------
1 / 1
___________________________________________________________________________________________________ BenchmarkTest.test_inference_no_model_no_architectures ___________________________________________________________________________________________________

self = <tests.test_benchmark.BenchmarkTest testMethod=test_inference_no_model_no_architectures>

    def test_inference_no_model_no_architectures(self):
        MODEL_ID = "sshleifer/tiny-gpt2"
        config = AutoConfig.from_pretrained(MODEL_ID)
        # set architectures equal to `None`
        config.architectures = None
        benchmark_args = PyTorchBenchmarkArguments(
            models=[MODEL_ID],
            training=True,
            inference=True,
            sequence_lengths=[8],
            batch_sizes=[1],
            multi_process=False,
        )
        benchmark = PyTorchBenchmark(benchmark_args, configs=[config])
>       results = benchmark.run()

tests/test_benchmark.py:114:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/transformers/benchmark/benchmark_utils.py:707: in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:676: in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark.py:69: in _inference_memory
    return self._measure_memory(_inference)
src/transformers/benchmark/benchmark.py:258: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-4' parent=1591 initial>, file = <_io.BytesIO object at 0x19b673a40>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
-------------------------------------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------------------------------------
1 / 1
__________________________________________________________________________________________________________ BenchmarkTest.test_inference_torchscript __________________________________________________________________________________________________________

self = <tests.test_benchmark.BenchmarkTest testMethod=test_inference_torchscript>

    def test_inference_torchscript(self):
        MODEL_ID = "sshleifer/tiny-gpt2"
        benchmark_args = PyTorchBenchmarkArguments(
            models=[MODEL_ID],
            training=False,
            inference=True,
            torchscript=True,
            sequence_lengths=[8],
            batch_sizes=[1],
            multi_process=False,
        )
        benchmark = PyTorchBenchmark(benchmark_args)
>       results = benchmark.run()

tests/test_benchmark.py:79:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/transformers/benchmark/benchmark_utils.py:707: in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:676: in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark.py:69: in _inference_memory
    return self._measure_memory(_inference)
src/transformers/benchmark/benchmark.py:258: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-5' parent=1591 initial>, file = <_io.BytesIO object at 0x1a390e9f0>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
-------------------------------------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------------------------------------
1 / 1
_________________________________________________________________________________________________________ BenchmarkTest.test_inference_with_configs __________________________________________________________________________________________________________

self = <tests.test_benchmark.BenchmarkTest testMethod=test_inference_with_configs>

    def test_inference_with_configs(self):
        MODEL_ID = "sshleifer/tiny-gpt2"
        config = AutoConfig.from_pretrained(MODEL_ID)
        benchmark_args = PyTorchBenchmarkArguments(
            models=[MODEL_ID],
            training=False,
            inference=True,
            sequence_lengths=[8],
            batch_sizes=[1],
            multi_process=False,
        )
        benchmark = PyTorchBenchmark(benchmark_args, configs=[config])
>       results = benchmark.run()

tests/test_benchmark.py:162:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/transformers/benchmark/benchmark_utils.py:707: in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:676: in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark.py:69: in _inference_memory
    return self._measure_memory(_inference)
src/transformers/benchmark/benchmark.py:258: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-6' parent=1591 initial>, file = <_io.BytesIO object at 0x1a391bd60>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
-------------------------------------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------------------------------------
1 / 1
_____________________________________________________________________________________________________________ BenchmarkTest.test_save_csv_files ______________________________________________________________________________________________________________

self = <tests.test_benchmark.BenchmarkTest testMethod=test_save_csv_files>

    def test_save_csv_files(self):
        MODEL_ID = "sshleifer/tiny-gpt2"
        with tempfile.TemporaryDirectory() as tmp_dir:
            benchmark_args = PyTorchBenchmarkArguments(
                models=[MODEL_ID],
                training=True,
                inference=True,
                save_to_csv=True,
                sequence_lengths=[8],
                batch_sizes=[1],
                inference_time_csv_file=os.path.join(tmp_dir, "inf_time.csv"),
                train_memory_csv_file=os.path.join(tmp_dir, "train_mem.csv"),
                inference_memory_csv_file=os.path.join(tmp_dir, "inf_mem.csv"),
                train_time_csv_file=os.path.join(tmp_dir, "train_time.csv"),
                env_info_csv_file=os.path.join(tmp_dir, "env.csv"),
                multi_process=False,
            )
            benchmark = PyTorchBenchmark(benchmark_args)
>           benchmark.run()

tests/test_benchmark.py:232:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/transformers/benchmark/benchmark_utils.py:707: in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:676: in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark.py:69: in _inference_memory
    return self._measure_memory(_inference)
src/transformers/benchmark/benchmark.py:258: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-7' parent=1591 initial>, file = <_io.BytesIO object at 0x1a39b9180>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
-------------------------------------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------------------------------------
1 / 1
______________________________________________________________________________________________________________ BenchmarkTest.test_trace_memory _______________________________________________________________________________________________________________

self = <tests.test_benchmark.BenchmarkTest testMethod=test_trace_memory>

    def test_trace_memory(self):
        MODEL_ID = "sshleifer/tiny-gpt2"

        def _check_summary_is_not_empty(summary):
            self.assertTrue(hasattr(summary, "sequential"))
            self.assertTrue(hasattr(summary, "cumulative"))
            self.assertTrue(hasattr(summary, "current"))
            self.assertTrue(hasattr(summary, "total"))

        with tempfile.TemporaryDirectory() as tmp_dir:
            benchmark_args = PyTorchBenchmarkArguments(
                models=[MODEL_ID],
                training=True,
                inference=True,
                sequence_lengths=[8],
                batch_sizes=[1],
                log_filename=os.path.join(tmp_dir, "log.txt"),
                log_print=True,
                trace_memory_line_by_line=True,
                multi_process=False,
            )
            benchmark = PyTorchBenchmark(benchmark_args)
>           result = benchmark.run()

tests/test_benchmark.py:261:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/transformers/benchmark/benchmark_utils.py:707: in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:676: in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark.py:69: in _inference_memory
    return self._measure_memory(_inference)
src/transformers/benchmark/benchmark.py:258: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-8' parent=1591 initial>, file = <_io.BytesIO object at 0x1a3a338b0>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
-------------------------------------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------------------------------------
1 / 1
-------------------------------------------------------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------------------------------------------------------
py3nvml not installed, we won't log GPU memory usage. Install py3nvml (pip install py3nvml) to use GPU memory tracing.
___________________________________________________________________________________________________ BenchmarkTest.test_train_encoder_decoder_with_configs ____________________________________________________________________________________________________

self = <tests.test_benchmark.BenchmarkTest testMethod=test_train_encoder_decoder_with_configs>

    def test_train_encoder_decoder_with_configs(self):
        MODEL_ID = "sshleifer/tinier_bart"
        config = AutoConfig.from_pretrained(MODEL_ID)
        benchmark_args = PyTorchBenchmarkArguments(
            models=[MODEL_ID],
            training=True,
            inference=True,
            sequence_lengths=[8],
            batch_sizes=[1],
            multi_process=False,
        )
        benchmark = PyTorchBenchmark(benchmark_args, configs=[config])
>       results = benchmark.run()

tests/test_benchmark.py:210:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/transformers/benchmark/benchmark_utils.py:707: in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:676: in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark.py:69: in _inference_memory
    return self._measure_memory(_inference)
src/transformers/benchmark/benchmark.py:258: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-9' parent=1591 initial>, file = <_io.BytesIO object at 0x1a3dce630>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
-------------------------------------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------------------------------------
1 / 1
____________________________________________________________________________________________________________ BenchmarkTest.test_train_no_configs _____________________________________________________________________________________________________________

self = <tests.test_benchmark.BenchmarkTest testMethod=test_train_no_configs>

    def test_train_no_configs(self):
        MODEL_ID = "sshleifer/tiny-gpt2"
        benchmark_args = PyTorchBenchmarkArguments(
            models=[MODEL_ID],
            training=True,
            inference=False,
            sequence_lengths=[8],
            batch_sizes=[1],
            multi_process=False,
        )
        benchmark = PyTorchBenchmark(benchmark_args)
>       results = benchmark.run()

tests/test_benchmark.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/transformers/benchmark/benchmark_utils.py:715: in run
    memory, train_summary = self.train_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:679: in train_memory
    return separate_process_wrapper_fn(self._train_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark.py:79: in _train_memory
    return self._measure_memory(_train)
src/transformers/benchmark/benchmark.py:258: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-10' parent=1591 initial>, file = <_io.BytesIO object at 0x1a3e422c0>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
-------------------------------------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------------------------------------
1 / 1
___________________________________________________________________________________________________________ BenchmarkTest.test_train_with_configs ____________________________________________________________________________________________________________

self = <tests.test_benchmark.BenchmarkTest testMethod=test_train_with_configs>

    def test_train_with_configs(self):
        MODEL_ID = "sshleifer/tiny-gpt2"
        config = AutoConfig.from_pretrained(MODEL_ID)
        benchmark_args = PyTorchBenchmarkArguments(
            models=[MODEL_ID],
            training=True,
            inference=False,
            sequence_lengths=[8],
            batch_sizes=[1],
            multi_process=False,
        )
        benchmark = PyTorchBenchmark(benchmark_args, configs=[config])
>       results = benchmark.run()

tests/test_benchmark.py:194:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/transformers/benchmark/benchmark_utils.py:715: in run
    memory, train_summary = self.train_memory(model_name, batch_size, sequence_length)
src/transformers/benchmark/benchmark_utils.py:679: in train_memory
    return separate_process_wrapper_fn(self._train_memory, self.args.do_multi_processing)(*args, **kwargs)
src/transformers/benchmark/benchmark.py:79: in _train_memory
    return self._measure_memory(_train)
src/transformers/benchmark/benchmark.py:258: in _measure_memory
    memory_bytes = measure_peak_memory_cpu(func)
src/transformers/benchmark/benchmark_utils.py:291: in measure_peak_memory_cpu
    mem_process.start()
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = <MemoryMeasureProcess name='MemoryMeasureProcess-11' parent=1591 initial>, file = <_io.BytesIO object at 0x1a3f06270>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'

/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
-------------------------------------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------------------------------------
1 / 1
====================================================================================================================== warnings summary ======================================================================================================================
venv/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:22
  /Users/sergiov/PycharmProjects/transformers/venv/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

tests/test_benchmark.py::BenchmarkTest::test_inference_torchscript
  /Users/sergiov/PycharmProjects/transformers/src/transformers/models/gpt2/modeling_gpt2.py:196: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
    attn_weights = attn_weights / (float(value.size(-1)) ** 0.5)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
================================================================================================================== short test summary info ===================================================================================================================
FAILED tests/test_benchmark.py::BenchmarkTest::test_inference_encoder_decoder_with_configs - AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'
FAILED tests/test_benchmark.py::BenchmarkTest::test_inference_no_configs - AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'
FAILED tests/test_benchmark.py::BenchmarkTest::test_inference_no_configs_only_pretrain - AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'
FAILED tests/test_benchmark.py::BenchmarkTest::test_inference_no_model_no_architectures - AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'
FAILED tests/test_benchmark.py::BenchmarkTest::test_inference_torchscript - AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'
FAILED tests/test_benchmark.py::BenchmarkTest::test_inference_with_configs - AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'
FAILED tests/test_benchmark.py::BenchmarkTest::test_save_csv_files - AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'
FAILED tests/test_benchmark.py::BenchmarkTest::test_trace_memory - AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'
FAILED tests/test_benchmark.py::BenchmarkTest::test_train_encoder_decoder_with_configs - AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'
FAILED tests/test_benchmark.py::BenchmarkTest::test_train_no_configs - AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'
FAILED tests/test_benchmark.py::BenchmarkTest::test_train_with_configs - AttributeError: Can't pickle local object 'measure_peak_memory_cpu.<locals>.MemoryMeasureProcess'
========================================================================================================= 11 failed, 2 skipped, 2 warnings in 11.71s =========================================================================================================
 

I'm seeing a similar error when following the [Benchmark documentation ](https://huggingface.co/transformers/benchmarks.html) while using Python 3.9.6. 

Test script executing documentation example:
```
import sys
import torch
import transformers
from transformers import PyTorchBenchmark, PyTorchBenchmarkArguments, BertConfig

print("Transformers Version: ", transformers.__version__)
print("PyTorch Version: ", torch.__version__)
print("Python Version: ", sys.version)

args = PyTorchBenchmarkArguments(models=["bert-base", "bert-384-hid", "bert-6-lay"], batch_sizes=[8], sequence_lengths=[8, 32, 128, 512])
config_base = BertConfig()
config_384_hid = BertConfig(hidden_size=384)
config_6_lay = BertConfig(num_hidden_layers=6)
benchmark = PyTorchBenchmark(args, configs=[config_base, config_384_hid, config_6_lay])
benchmark.run()
```

Output:
```
python test_benchmark.py 
Transformers Version:  4.12.3
PyTorch Version:  1.10.0
Python Version:  3.9.6 (default, Sep  8 2021, 11:57:49) 
[Clang 12.0.5 (clang-1205.0.22.11)]
1 / 3
Traceback (most recent call last):
  File "/Users/daubner/dev/email-zoning/test_benchmark.py", line 15, in <module>
    benchmark.run()
  File "/Users/daubner/.venv/benchmark/lib/python3.9/site-packages/transformers/benchmark/benchmark_utils.py", line 707, in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
  File "/Users/daubner/.venv/benchmark/lib/python3.9/site-packages/transformers/benchmark/benchmark_utils.py", line 676, in inference_memory
    return separate_process_wrapper_fn(self._inference_memory, self.args.do_multi_processing)(*args, **kwargs)
  File "/Users/daubner/.venv/benchmark/lib/python3.9/site-packages/transformers/benchmark/benchmark_utils.py", line 101, in multi_process_func
    p.start()
  File "/Users/daubner/.pyenv/versions/3.9.6/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/daubner/.pyenv/versions/3.9.6/lib/python3.9/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/daubner/.pyenv/versions/3.9.6/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/daubner/.pyenv/versions/3.9.6/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/daubner/.pyenv/versions/3.9.6/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/daubner/.pyenv/versions/3.9.6/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/daubner/.pyenv/versions/3.9.6/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'separate_process_wrapper_fn.<locals>.multi_process_func.<locals>.wrapper_func'
```

Appears to be caused by a difference in the `multiprocessing` library from Python 3.7 to 3.9. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hey, bumping this to keep it alive - it's something we're definitely aware of but it's not a priority right now, given that tests seem to work in the CI. If anyone can investigate it and submit a PR that would be great, if not we'll get to it as soon as we can!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I found time to get around to this but could not reproduce - `pytest test_benchmark.py` and `pytest test_tf_benchmark.py` work fine for me on Python 3.9.7 with the current master version of Transformers. Possibly the issue was resolved by a patch to transformers or Python in the meantime?