Can't pickle local object using the finetuning example.
Author: Normand-1024
Pull request: no
Labels: name: wontfix

I was testing out the finetuning example from the repo:

`python run_lm_finetuning.py --train_data_file="finetune-output/KantText.txt" --output_dir="finetune-output/hugkant" --model_type=gpt2 --model_name_or_path=gpt2 --do_train --block_size=128`

While saving the checkpoint, it gives the following error:

```

Traceback (most recent call last):
  File "run_lm_finetuning.py", line 790, in <module>
    main()
  File "run_lm_finetuning.py", line 740, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 398, in train
    torch.save(scheduler.state_dict(), os.path.join(output_dir, "scheduler.pt"))
  File "D:\Software\Python\lib\site-packages\torch\serialization.py", line 209, in save
    return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
  File "D:\Software\Python\lib\site-packages\torch\serialization.py", line 134, in _with_file_like
    return body(f)
  File "D:\Software\Python\lib\site-packages\torch\serialization.py", line 209, in <lambda>
    return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
  File "D:\Software\Python\lib\site-packages\torch\serialization.py", line 282, in _save
    pickler.dump(obj)
AttributeError: Can't pickle local object 'get_linear_schedule_with_warmup.<locals>.lr_lambda'
```

Comments:
Do you mind specifying which versions of everything you're using, as detailed in the [bug report issue template](https://github.com/huggingface/transformers/issues/new/choose)?

Hi @Normand-1024 

were you able to fix this error?
as i am getting the same error while trying to run glue task (QQP) but works fine when i run MRPC.

Hi,

I was able to get rid of this error by upgrading the torch version.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Still having this issue with transformers `2.11.0`.

@lucadiliello did you maange to fix it?

Not yet. A solution would be to use `dill` instead of `pickle`... but I'm not sure how to do it.

Getting same error,
Not sure how to fix this error.

same error, with all newest version

I solved by reimplementing all the schedulers without lambda functions. [here](https://github.com/iKernels/transformers-lightning) I published many schedulers.

Same error with all newest version too.

![image](https://user-images.githubusercontent.com/44800122/115481664-afdd6300-a27f-11eb-9a6e-77b5ed4afe02.png)


UP. 
Is it a package version related issue?

Having the same issue: 
`Can't pickle local object 'get_linear_schedule_with_warmup.<locals>.lr_lambda'`

For running the example scripts passing `--no_multi_process` solved it for me.

I haven't looked into the huggingface code yet but I could imagine that [this](https://stackoverflow.com/questions/52265120/python-multiprocessing-pool-attributeerror) is the bug here. I think it only shows up when `spawn` instead of `fork` is used to create new processes, which is why the developers might have missed it.

I set the `gpus=1`, and it works. 

Well, this seems that it is a local object that can not be forked, you may define it at each forked process. This may work well. However, somebody should fix it.