Transformers tokenizer pickling issue using hydra and submitit_slurm
Author: asishgeek
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.8.2
- Platform: Linux-5.4.0-1051-aws-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.9.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
- 
### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 -->
@LysandreJik

## Information

Model I am using (Bert, XLNet ...): t5

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Run the script using command:

    python hf_hydra.py hydra/launcher=submitit_slurm -m

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

Code (hf_hydra.py):

    import hydra
    import logging
    # from transformers import AutoTokenizer
    import transformers
    
    
    @hydra.main(config_path=None)
    def main(cfg):
        logger = logging.getLogger(__name__)
        # tokenizer = AutoTokenizer.from_pretrained("t5-small")
        tokenizer = transformers.T5Tokenizer.from_pretrained("t5-small")
        logger.info(f"vocab size: {tokenizer.vocab_size}")
    
    
    if __name__ == '__main__':
        main()


Using AutoTokenizer works but using T5Tokenizer fails with the following error.

Traceback (most recent call last):
  File "/data/home/aghoshal/miniconda/lib/python3.8/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
    return func()
  File "/data/home/aghoshal/miniconda/lib/python3.8/site-packages/hydra/_internal/utils.py", line 376, in <lambda>
    lambda: hydra.multirun(
  File "/data/home/aghoshal/miniconda/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 139, in multirun
    ret = sweeper.sweep(arguments=task_overrides)
  File "/data/home/aghoshal/miniconda/lib/python3.8/site-packages/hydra/_internal/core_plugins/basic_sweeper.py", line 157, in sweep
    results = self.launcher.launch(batch, initial_job_idx=initial_job_idx)
  File "/data/home/aghoshal/miniconda/lib/python3.8/site-packages/hydra_plugins/hydra_submitit_launcher/submitit_launcher.py", line 145, in launch
    jobs = executor.map_array(self, *zip(*job_params))
  File "/data/home/aghoshal/miniconda/lib/python3.8/site-packages/submitit/core/core.py", line 631, in map_array
    return self._internal_process_submissions(submissions)
  File "/data/home/aghoshal/miniconda/lib/python3.8/site-packages/submitit/auto/auto.py", line 213, in _internal_process_submissions
    return self._executor._internal_process_submissions(delayed_submissions)
  File "/data/home/aghoshal/miniconda/lib/python3.8/site-packages/submitit/slurm/slurm.py", line 313, in _internal_process_submissions
    return super()._internal_process_submissions(delayed_submissions)
  File "/data/home/aghoshal/miniconda/lib/python3.8/site-packages/submitit/core/core.py", line 749, in _internal_process_submissions
    delayed.dump(pickle_path)
  File "/data/home/aghoshal/miniconda/lib/python3.8/site-packages/submitit/core/utils.py", line 136, in dump
    cloudpickle_dump(self, filepath)
  File "/data/home/aghoshal/miniconda/lib/python3.8/site-packages/submitit/core/utils.py", line 240, in cloudpickle_dump
    cloudpickle.dump(obj, ofile, pickle.HIGHEST_PROTOCOL)
  File "/data/home/aghoshal/miniconda/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py", line 55, in dump
    CloudPickler(
  File "/data/home/aghoshal/miniconda/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py", line 563, in dump
    return Pickler.dump(self, obj)
TypeError: cannot pickle '_LazyModule' object

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Job should run and print the vocab size.

Comments:
This has been solved in v4.9, you should upgrade to the latest version of Transformers.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.