The squad processor's multi-threading crashes the script / causes large models to reload every call
Author: Avlyssna
Pull request: no
Labels: name: wontfix

## Environment info
     
- `transformers` version: 3.0.2
- Platform: Windows-10-10.0.17763-SP0
- Python version: 3.6.8
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: NA

### Who can help
@mfuntowicz

## Information
When using the `question-answering` pipeline, my script is run multiple times due to the thread pool within `squad_convert_examples_to_features`. In combination with other large models that take minutes to load, this causes them to reload every time the pipeline is called - or error outright if the code is not encapsulated. I've temporarily patched the `transformers\data\processors\squad.py` file on my end, forcing it to run in the current thread rather than multi-threading:

```python
    # with Pool(threads, initializer=squad_convert_example_to_features_init, initargs=(tokenizer,)) as p:
    #     annotate_ = partial(
    #         squad_convert_example_to_features,
    #         max_seq_length=max_seq_length,
    #         doc_stride=doc_stride,
    #         max_query_length=max_query_length,
    #         is_training=is_training,
    #     )
    #     features = list(
    #         tqdm(
    #             p.imap(annotate_, examples, chunksize=32),
    #             total=len(examples),
    #             desc="convert squad examples to features",
    #             disable=not tqdm_enabled,
    #         )
    #     )

    squad_convert_example_to_features_init(tokenizer)

    for example in examples:
        features.append(squad_convert_example_to_features(
            example,
            max_seq_length=max_seq_length,
            doc_stride=doc_stride,
            max_query_length=max_query_length,
            is_training=is_training
        ))
```

I'm wondering if there's a better solution here, though. Is this a bug? Or maybe an option could added in to allow single-threaded processing of the `examples`? I may be doing something wrong on my end too; not sure.

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. For a simplified example, run the following python script from terminal:
```python
#!/usr/bin/env python3
from transformers import pipeline

print('The script has been imported.')
nlp = pipeline('question-answering', framework='pt')
print(nlp(question='Who walked on the moon?', context='Niel Armstrong walked on the moon.'))
```

```
(environment) C:\Users\Dennis\Desktop\New folder>python runner.py
The script has been imported.
The script has been imported.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:\users\dennis\appdata\local\programs\python\python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
Traceback (most recent call last):
    exitcode = _main(fd)
  File "runner.py", line 6, in <module>
  File "c:\users\dennis\appdata\local\programs\python\python36\lib\multiprocessing\spawn.py", line 114, in _main
    print(nlp(question='Who walked on the moon?', context='Niel Armstrong walked on the moon.'))
    prepare(preparation_data)
  File "C:\Users\Dennis\Desktop\New folder\environment\lib\site-packages\transformers\pipelines.py", line 1264, in __call__
  File "c:\users\dennis\appdata\local\programs\python\python36\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "c:\users\dennis\appdata\local\programs\python\python36\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    for example in examples
  File "C:\Users\Dennis\Desktop\New folder\environment\lib\site-packages\transformers\pipelines.py", line 1264, in <listcomp>
    run_name="__mp_main__")
  File "c:\users\dennis\appdata\local\programs\python\python36\lib\runpy.py", line 263, in run_path
    for example in examples
    pkg_name=pkg_name, script_name=fname)
  File "C:\Users\Dennis\Desktop\New folder\environment\lib\site-packages\transformers\data\processors\squad.py", line 325, in squad_convert_examples_to_features
  File "c:\users\dennis\appdata\local\programs\python\python36\lib\runpy.py", line 96, in _run_module_code
    with Pool(threads, initializer=squad_convert_example_to_features_init, initargs=(tokenizer,)) as p:
    mod_name, mod_spec, pkg_name, script_name)
  File "c:\users\dennis\appdata\local\programs\python\python36\lib\multiprocessing\context.py", line 119, in Pool
  File "c:\users\dennis\appdata\local\programs\python\python36\lib\runpy.py", line 85, in _run_code
    context=self.get_context())
  File "c:\users\dennis\appdata\local\programs\python\python36\lib\multiprocessing\pool.py", line 174, in __init__
    exec(code, run_globals)
  File "C:\Users\Dennis\Desktop\New folder\runner.py", line 6, in <module>
    self._repopulate_pool()
    print(nlp(question='Who walked on the moon?', context='Niel Armstrong walked on the moon.'))
  File "c:\users\dennis\appdata\local\programs\python\python36\lib\multiprocessing\pool.py", line 239, in _repopulate_pool
  File "C:\Users\Dennis\Desktop\New folder\environment\lib\site-packages\transformers\pipelines.py", line 1264, in __call__
    w.start()
  File "c:\users\dennis\appdata\local\programs\python\python36\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "c:\users\dennis\appdata\local\programs\python\python36\lib\multiprocessing\context.py", line 322, in _Popen
    for example in examples
  File "C:\Users\Dennis\Desktop\New folder\environment\lib\site-packages\transformers\pipelines.py", line 1264, in <listcomp>
    return Popen(process_obj)
  File "c:\users\dennis\appdata\local\programs\python\python36\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
    for example in examples
  File "c:\users\dennis\appdata\local\programs\python\python36\lib\multiprocessing\reduction.py", line 60, in dump
  File "C:\Users\Dennis\Desktop\New folder\environment\lib\site-packages\transformers\data\processors\squad.py", line 325, in squad_convert_examples_to_features
    ForkingPickler(file, protocol).dump(obj)
    with Pool(threads, initializer=squad_convert_example_to_features_init, initargs=(tokenizer,)) as p:
BrokenPipeError: [Errno 32] Broken pipe
  File "c:\users\dennis\appdata\local\programs\python\python36\lib\multiprocessing\context.py", line 119, in Pool
    context=self.get_context())
  File "c:\users\dennis\appdata\local\programs\python\python36\lib\multiprocessing\pool.py", line 174, in __init__
    self._repopulate_pool()
  File "c:\users\dennis\appdata\local\programs\python\python36\lib\multiprocessing\pool.py", line 239, in _repopulate_pool
    w.start()
  File "c:\users\dennis\appdata\local\programs\python\python36\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "c:\users\dennis\appdata\local\programs\python\python36\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "c:\users\dennis\appdata\local\programs\python\python36\lib\multiprocessing\popen_spawn_win32.py", line 33, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "c:\users\dennis\appdata\local\programs\python\python36\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
    _check_not_importing_main()
  File "c:\users\dennis\appdata\local\programs\python\python36\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
    is not going to be frozen to produce an executable.''')
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
```

## Expected behavior
The script doesn't crash / reload the script every pipeline call.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Did anyone find the solution to this?