issue with evaluation of seq2seq_trainer.py on multiple gpus
Author: rabeehk
Pull request: no

## Environment info
- `transformers` version: 3.5.1 
- Platform: Linux 
- Python version: 3.7
- PyTorch version (GPU?): 1.7
- Tensorflow version (GPU?): - 
- Using GPU in script?: yes 
- Using distributed or parallel set-up in script?: yes 

### Who can help

 Trainer: @sgugger

## Information and commands to reproduce

I am running seq2seq trainer model on multiple GPUS. The problem arises during evaluation, here are my modified seq2seq_trainer.py codes and how to reproduce the error: 

```
git clone git@github.com:rabeehk/seq2seq.git
python setup.py develop
cd seq2seq
python -m torch.distributed.launch --nproc_per_node=4 --master_port=9918  finetune_t5_trainer.py temp_configs/mp-lr-3e-02-r-8-l-true.json 
```

here is the error I get during the evaluation:

`file outputs/mixture1/meta-adapters-task-projector-new_sampler-num-gpus-4/mp-lr-3e-02-r-8-l-true/config.json not found
`

## Bug description 
I figured out in finetune_trainer.py if a user load config and trained model again during the evaluation, this operation results in processes not being able to find the config file, showing this is not multi-gpu safe, I realized if I add `trainer.is_world_process_zero:` beforehand and reload model and config in evaluation part, this resolves the issue, could you please comment on this and assist with this issue? I think this is a bug if the user cannot load the model in the evaluation part in multiple-gpus properly. Thanks.

## Full error stack.
 ```
I 1222 15:38:32.129721  2784 main shadow.py:122] > Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py", line 388, in get_config_dict
file outputs/mixture1/meta-adapters-task-projector-new_sampler-num-gpus-4/mp-lr-3e-02-r-8-l-true/config.json not found
I 1222 15:38:32.129757  2784 main shadow.py:122] > Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py", line 388, in get_config_dict
                                                   Traceback (most recent call last):
I 1222 15:38:32.129800  2784 main shadow.py:122] > 
I 1222 15:38:32.130028  2784 main shadow.py:122] >     local_files_only=local_files_only,
I 1222 15:38:32.130069  2784 main shadow.py:122] >   File "/usr/local/lib/python3.6/dist-packages/transformers/file_utils.py", line 962, in cached_path
I 1222 15:38:32.130314  2784 main shadow.py:122] >     raise EnvironmentError("file {} not found".format(url_or_filename))
I 1222 15:38:32.130367  2784 main shadow.py:122] > OSError: file outputs/mixture1/meta-adapters-task-projector-new_sampler-num-gpus-4/mp-lr-3e-02-r-8-l-true/config.json not found

I 1222 15:38:32.130448  2784 main shadow.py:122] > During handling of the above exception, another exception occurred:
I 1222 15:38:32.130505  2784 main shadow.py:122] > 
I 1222 15:38:32.130557  2784 main shadow.py:122] > OSError: file outputs/mixture1/meta-adapters-task-projector-new_sampler-num-gpus-4/mp-lr-3e-02-r-8-l-true/config.json not found
I 1222 15:38:32.130720  2784 main shadow.py:122] > Traceback (most recent call last):
OSError: file outputs/mixture1/meta-adapters-task-projector-new_sampler-num-gpus-4/mp-lr-3e-02-r-8-l-true/config.json not found
I 1222 15:38:32.130823  2784 main shadow.py:122] >   File "./finetune_t5_trainer.py", line 328, in <module>
I 1222 15:38:32.130858  2784 main shadow.py:122] > 
I 1222 15:38:32.130903  2784 main shadow.py:122] > During handling of the above exception, another exception occurred:
I 1222 15:38:32.130938  2784 main shadow.py:122] > 
I 1222 15:38:32.130976  2784 main shadow.py:122] > Traceback (most recent call last):
  File "./finetune_t5_trainer.py", line 207, in main
    main()
  File "/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py", line 347, in from_pretrained
    cache_dir=model_args.cache_dir)
  File "/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py", line 347, in from_pretrained
100%|??????????| 2100/2100 [36:51<00:00,  1.05s/it]
I 1222 15:38:32.131012  2784 main shadow.py:122] >     cache_dir=model_args.cache_dir)
I 1222 15:38:32.131047  2784 main shadow.py:122] >   File "/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py", line 347, in from_pretrained
I 1222 15:38:32.131081  2784 main shadow.py:122] >     config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
I 1222 15:38:32.131301  2784 main shadow.py:122] >   File "/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py", line 400, in get_config_dict
I 1222 15:38:32.131337  2784 main shadow.py:122] >     raise EnvironmentError(msg)
I 1222 15:38:32.131372  2784 main shadow.py:122] > OSError: Can't load config for 'outputs/mixture1/meta-adapters-task-projector-new_sampler-num-gpus-4/mp-lr-3e-02-r-8-l-true'. Make sure that:
I 1222 15:38:32.131432  2784 main shadow.py:122] > - 'outputs/mixture1/meta-adapters-task-projector-new_sampler-num-gpus-4/mp-lr-3e-02-r-8-l-true' is a correct model identifier listed on 'https://huggingface.co/models'
I 1222 15:38:32.131615  2784 main shadow.py:122] >     raise EnvironmentError(msg)
I 1222 15:38:32.131670  2784 main shadow.py:122] > OSError: Can't load config for 'outputs/mixture1/meta-adapters-task-projector-new_sampler-num-gpus-4/mp-lr-3e-02-r-8-l-true'. Make sure that:
```


Comments:
Hello @rabeehk,

We sadly cannot fix issues of different repos, such as `rabeehk/seq2seq.git` - this is too time-consuming and not really our responsibility. We're happy to assist if you could provide a **short, precise, and complete** code snippet that is based on Transformers `Seq2SeqTrainer` only.

@rabeehk, I think you may have not considered that open source projects are not a help desk. If you are going to continue in the same fashion you will not get any answers at all.

Many people ask for help but you need to think how to ask for help so that it's easy for the developers to quickly understand what is going on, reproduce the problem and solve what needs to be solved. But if you dump 1000 line logs and say help me to fix this without investigating it first yourself you will not get anywhere here.

For example, in your 1000 line log dump in OP if you look closely you will see that the error is on your side since it tells you:
```
outputs/mixture1/meta-adapters-task-projector-new_sampler-num-gpus-4/mp-lr-3e-02-r-8-l-true/config.json not found
```
i.e. your setup is broken.

So you didn't really study the problem and yet want us to do this for you.

That's said I personally will not do it again, so please don't tag me unless it's related to what I'm working on and you found a bug in the code I wrote or maintain.

Also tagging multiple developers out of context is frowned upon - you tagged me on this issue:

> Who can help
> FSMT: @stas00

what does it have to do with FSMT? 

The tagging info is to help users to direct their questions to the right developers who are maintainers of certain domains. They can then decide at their own discretion to tag other developers if they feel it'd help move the issue forward. If you tag multiple people out of context you will gain no support. 

If you are not willing to invest energy and time into investigating the problems you encounter and forming quality questions, please consider hiring someone who will be willing to answer the multitude of your questions and sort things out for you. Perhaps ask at the forums if someone is willing to work with you professionally where you pay them for the services provided.

I hope this comment has been useful and trust you will find a way to receive the support you need.


Yes @stas00 is completely right. @rabeehk your comments are borderline spammy. 

We try to help as much as possible but you also need to put in the work so the community can actually help you efficiently.

Hi Stephan, Hi Julien
please find my responses below:

On Wed, Dec 23, 2020 at 6:29 PM Stas Bekman <notifications@github.com>
wrote:

> rabeehk, I think you may have not considered that open source projects are
> not a help desk. If you are going to continue in the same fashion you will
> not get any answers at all.
>
Sorry if this looks like a spam to you, but I really still think this was a
bug, if you try to load the model twice inside the finetune_trainer.py in
evaluation part, which is something the user might well need when one wants
to apply more chnages to the trained model before evaluation, you would see
this is not multi-process safe resulting in the bug I reported.

> Many people ask for help but you need to think how to ask for help so that
> it's easy for the developers to quickly understand what is going on,
> reproduce the problem and solve what needs to be solved. But if you dump
> 1000 line logs and say help me to fix this without investigating it first
> yourself you will not get anywhere here.
>
Sorry I thought providing full logs help, if this is not sure, I would not
provide the full logs, I still included the one line error message above
these logs, I did investigate the issue myself, and I realized this is not
multi-process safe as I mentioned.

> For example, in your 1000 line log dump in OP if you look closely you will
> see that the error is on your side since it tells you:
>
> outputs/mixture1/meta-adapters-task-projector-new_sampler-num-gpus-4/mp-lr-3e-02-r-8-l-true/config.json not found
>
> no the setup is not broken, the files are there as I said, please read my
bug report carefully,  this is the result of the bug I said.

> i.e. your setup is broken.
>
> So you didn't really study the problem and yet want us to do this for you.
>
> not really, I investigated it for hours and hours in fact.

> That's said I personally will not do it again, so please don't tag me
> unless it's related to what I'm working on and you found a bug in the code
> I wrote or maintain.
>
> Sure, I was thinking you are working on seq2seq from various updates on
this, sorry for the mistake,

> Also tagging multiple developers out of context is frowned upon - you
> tagged me on this issue:
>
> Who can help
> FSMT: @stas00 <https://github.com/stas00>
>
> what does it have to do with FSMT?
>
sorry for the mistake, I explained this above, I was really thinking you
are working on seq2seq and though this is relevant.

> The tagging info is to help users to direct their questions to the right
> developers who are maintainers of certain domains. They can then decide to
> tag other developers if they feel it'd help the issue along. If you tag
> multiple people out of context you will gain no support.
>
> I was really mistaken thinking  this is relevant.

> If you are not willing to invest energy and time into investigating the
> problems you encounter and forming quality questions, please consider
> hiring someone who will be willing to answer the multitude of your
> questions and sort things out for you. Perhaps ask at the forums if someone
> is willing to work with you professionally where you pay them for the
> services provided.
>
No, this is not correct, I did spent hours and hours on this, to me this is
still a bug.

> I hope this comment has been useful and trust you will find a way to
> receive the support you need.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/9280#issuecomment-750399462>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABP4ZCBMPNG4IN2NE5DGRPDSWISH7ANCNFSM4VG3SIFA>
> .
>


I also should say putting comments like your last paragraph @stas00  is inappropripate. No matter I beleive this is a bug you think this is a spam, no matter in which position you are, no matter I mistakenly thought providing full logs helps,  please behave people with respect. I still believe this is a bug.

On Wed, Dec 23, 2020, 8:43 PM Rabeeh Karimi <rabeeh.k68@gmail.com> wrote:

> Hi Stephan, Hi Julien
> please find my responses below:
>
> On Wed, Dec 23, 2020 at 6:29 PM Stas Bekman <notifications@github.com>
> wrote:
>
>> rabeehk, I think you may have not considered that open source projects
>> are not a help desk. If you are going to continue in the same fashion you
>> will not get any answers at all.
>>
> Sorry if this looks like a spam to you, but I really still think this was
> a bug, if you try to load the model twice inside the finetune_trainer.py in
> evaluation part, which is something the user might well need when one wants
> to apply more chnages to the trained model before evaluation, you would see
> this is not multi-process safe resulting in the bug I reported.
>
>> Many people ask for help but you need to think how to ask for help so
>> that it's easy for the developers to quickly understand what is going on,
>> reproduce the problem and solve what needs to be solved. But if you dump
>> 1000 line logs and say help me to fix this without investigating it first
>> yourself you will not get anywhere here.
>>
> Sorry I thought providing full logs help, if this is not sure, I would not
> provide the full logs, I still included the one line error message above
> these logs, I did investigate the issue myself, and I realized this is not
> multi-process safe as I mentioned.
>
>> For example, in your 1000 line log dump in OP if you look closely you
>> will see that the error is on your side since it tells you:
>>
>> outputs/mixture1/meta-adapters-task-projector-new_sampler-num-gpus-4/mp-lr-3e-02-r-8-l-true/config.json not found
>>
>> no the setup is not broken, the files are there as I said, please read my
> bug report carefully,  this is the result of the bug I said.
>
>> i.e. your setup is broken.
>>
>> So you didn't really study the problem and yet want us to do this for you.
>>
>> not really, I investigated it for hours and hours in fact.
>
>> That's said I personally will not do it again, so please don't tag me
>> unless it's related to what I'm working on and you found a bug in the code
>> I wrote or maintain.
>>
>> Sure, I was thinking you are working on seq2seq from various updates on
> this, sorry for the mistake,
>
>> Also tagging multiple developers out of context is frowned upon - you
>> tagged me on this issue:
>>
>> Who can help
>> FSMT: @stas00 <https://github.com/stas00>
>>
>> what does it have to do with FSMT?
>>
> sorry for the mistake, I explained this above, I was really thinking you
> are working on seq2seq and though this is relevant.
>
>> The tagging info is to help users to direct their questions to the right
>> developers who are maintainers of certain domains. They can then decide to
>> tag other developers if they feel it'd help the issue along. If you tag
>> multiple people out of context you will gain no support.
>>
>> I was really mistaken thinking  this is relevant.
>
>> If you are not willing to invest energy and time into investigating the
>> problems you encounter and forming quality questions, please consider
>> hiring someone who will be willing to answer the multitude of your
>> questions and sort things out for you. Perhaps ask at the forums if someone
>> is willing to work with you professionally where you pay them for the
>> services provided.
>>
> No, this is not correct, I did spent hours and hours on this, to me this
> is still a bug.
>
>> I hope this comment has been useful and trust you will find a way to
>> receive the support you need.
>>
>> —
>> You are receiving this because you were mentioned.
>> Reply to this email directly, view it on GitHub
>> <https://github.com/huggingface/transformers/issues/9280#issuecomment-750399462>,
>> or unsubscribe
>> <https://github.com/notifications/unsubscribe-auth/ABP4ZCBMPNG4IN2NE5DGRPDSWISH7ANCNFSM4VG3SIFA>
>> .
>>
>


The point we are trying to communicate is that you need to review how you communicate, @rabeehk. Your communications come across as too much and too indiscriminate.

I'm totally accepting that you might be unaware of what is expected in good communications and perhaps HuggingFace needs to have a guidelines document at how users can ask for help in the most efficient way for all involved parties.

As of this moment I'd happy to invest a bit of my free time to support you to find a way for you to become an asset to this community and not an annoyance.  If you are willing to listen and take action:

1. Anybody looking at your first post will have an urge to flee - it's scary in its length and most people will not even try to understand what could be a very valid issue.

   So you need to edit the first post to remove any information that's not pertaining to the issue at hand. e.g. all those Download xx% logs are totally useless. You're saying you are attaching the traceback, but you're attaching the full log. I accept that you might have not known that.

   Attaching a full log can be helpful if it's done as an attachment, a link to a paste.bin or at the very least if you enclosed it in:

```
<details>
<summary>Full log</summary>
<pre>

many
lines 
go
here

</pre>
</details>
```
   Here is an example of the outcome:

<details>
<summary>Full log</summary>
<pre>
many
lines 
go
here
</pre>
</details>

2. As @patrickvonplaten replied to you, you can't ask someone to go into your repository and figure out what you may have done. The code is already very complex and unless there is an easy way to do a diff and it's a small diff, nobody has the time to investigate. So you need to spend time to find a way to reproduce the problem in a minimal example which should introduce no more than a few lines of code change-wise (of course, there are exceptions, but this is more of  norm).

   Usually the best way is to just show the relevant backtrace (in DDP just one of them, as each process will dump a copy), the command line and then ask if there is anything else that you could supply to help the developer reproduce the problem.

3. Try to use the latest official version. We have no resources to go and debug older revisions, which could easily have bugs that have been fixed in the latest released version.

   I understand that this is not always possible. But this is the best way if it fits.

4. Most of the time you can't ask to test with your data, since we don't have your data. So either you should use some existing dataset supported by HF datasets or you need to have the code that generates a small sample on the fly.

5. Do not tag multiple people on the issue unless you know this is expected, either because you asked them and they gave you an explicit permission or the Issue template instructs you to do so. Having someone help you like I'm doing now is not an invitation to tag that person in the future on all your issues.

   I can see why you chose to tag me by looking at seq2seq commits, and while I made a few small changes in seq2seq recently, it just happened to be so because I was working on something totally unrelated and there were some changes that were required for me to proceed. But I'm not in charge of that domain.

   Remember that every time you tag someone, they get a notification and you're taking their time w/o their permission. Please be sensitive to that.

6. Use the edit button. Delete and merge multiple comments into one if nobody followed up yet. As you merge them edit them to be coherent. Use bullets and items if it makes sense.

   I know my first comment version almost always comes out with typos and can be incoherent or too verbose. If you look at my comments' history I often make a ton of re-edits, since I want to make sure my communication is as clear as possible (and I know I myself can be too verbose, mea culpa). 
   
---------------

The key message of this comment is that when you ask for support you are given a tiny sliver of developer's time and you need to quickly communicate the essentials of the issue at hand. You're not expected to be born with that knowledge.  You're not expected to be perfect at it. If I may recommend - learn from issues posted by other people - see which issues get responses and which are ignored - learn what the posters who did get responses did right. It's a simple pattern matching with some patience.

There is no harm in asking: "look, I have all these questions and I don't know how to ask them in the best possible way. Can someone help?"

Perhaps, you need to find a mentor in the community at the forums, by asking if someone can support you to help you find a way to file good issues.

When you tune up your communications then the developers of this fabulous project will be more than happy to address and resolve the issues you raise. This skill, of course, will help you at any other open source project.

Please let me know if you found this helpful. And perhaps if you'd like to continue this discussion because you need further clarifications let's go to the forums https://discuss.huggingface.co/ and leave the Issues section alone for now. You have my permission to tag me (just @stas) in the forums for this particular discussion if you think it'd be helpful to you.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.