Fix resume_from_checkpoint for deepspeed
Author: mosheber
Pull request: yes

@stas00  This fixes the resume_from_checkpoint for deepspeed, by ensuring that the deepspeed engine is the one to load the checkpoint.

# What does this PR do?

It disables the regular load_from_checkpoint, and allowing it to go to the deepspeed engine instead.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer: @stas00, Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_21735). All of your documentation changes will be reflected on that endpoint.

Hi @mosheber! The CI is not triggered. It seems there is an issue with your CircleCI permissions, the tests won't run.
Could you first try refreshing your permissions as shown [here](https://support.circleci.com/hc/en-us/articles/360048210711-How-to-Refresh-User-Permissions-)?

Thank you, and let me know if the CI could be triggered after this :-)

> 

@ydshieh @sgugger , thank you for approving! For some reason the CircleCI still wont run properly, I tried logging out, revoking, logging back in, refreshing, yet to avail. Could you perform the merge on your end? Or perhaps trigger the CI run? 

Hi @mosheber Thank you for trying. I will push an empty commit to your PR branch to trigger CI - are you OK with it?

Well, I think one more step to follow from your side: (as I am not able to trigger even with a push)


Could you check if you are following huggingface/transformers instead of your own fork. You can check it at this link
https://app.circleci.com/projects/project-dashboard/github/mosheber/

If you are following your own fork, you have to unfollow it, and follow `huggingface/transformers` instead.

> If a user submits a pull request to your repository from a fork, but no pipeline is triggered, then the user most likely is following a project fork on their personal account rather than the project itself of CircleCI, causing the jobs to trigger under the user’s personal account and not the organization account. To resolve this issue, have the user unfollow their fork of the project on CircleCI and instead follow the source project. This will trigger their jobs to run under the organization when they submit pull requests.


https://circleci.com/docs/oss/#build-pull-requests-from-forked-repositories

If you are OK, I can also fork your PR, create a new one, but add your name as a contributor then merge the new PR. This might be easier.

> If you are OK, I can also fork your PR, create a new one, but add your name as a contributor then merge the new PR. This might be easier.

This seems to be the easiest approach, lets go with that

well, while I pushed to `huggingface/transformers`, the CI here is triggered ... let's see.

> well, while I pushed to `huggingface/transformers`, the CI here is triggered ... let's see.

Looks like all checks have passed, thanks! Merging should now be possible 

> > well, while I pushed to `huggingface/transformers`, the CI here is triggered ... let's see.
> 
> Looks like all checks have passed, thanks! Merging should now be possible

Yeah, but see my comment

https://github.com/huggingface/transformers/pull/21735#discussion_r1117246877

> Looks like all checks have passed, thanks! Merging should now be possible

But it's not ready. Please revisit: https://github.com/huggingface/transformers/pull/21735#discussion_r1116267795

> > Looks like all checks have passed, thanks! Merging should now be possible
> 
> But it's not ready. Please revisit: [#21735 (comment)](https://github.com/huggingface/transformers/pull/21735#discussion_r1116267795)

Sure thing, I removed it and changed the elif

Thanks Moshe, I need to run offline tests since deepspeed tests don't run on live CI (need gpus), and will merge once all is green there.

@ydshieh - do you by chance have any idea why CI isn't running? This time it appears to be some other problem than the original one we discussed in this issue. Thank you!

Moshe, CircleCI doesn't like something about your CircleCI account settings. Since there is nothing I can do about it and I don't want this to drag forever I've recreated your PR here https://github.com/huggingface/transformers/pull/21798 - so let's finish it there. Thank you.


> Moshe, CircleCI doesn't like something about your CircleCI account settings. Since there is nothing I can do about it and I don't want this to drag forever I've recreated your PR here #21798 - so let's finish it there. Thank you.

No problem. Just in case, I also tried to trigger another CI run after unfollowing the project as suggested here:

https://support.circleci.com/hc/en-us/articles/360008097173-Troubleshooting-why-pull-requests-are-not-triggering-jobs-on-my-organization-

Maybe it will help too

Thank you for digging this up, Moshe. The relevant quote seems to be:

> If you're following the fork instead of the upstream repo
> 
> A user who submits a pull request to your repository from a fork, but no pipeline is triggered with the pull request. This can happen when the user is following the project fork on their personal account rather than the project itself on CircleCI.
> 
> This will cause the jobs to trigger under the user's personal account. If the user is following a fork of the repository on CircleCI, we will only build on that fork and not the parent, so the parent’s PR will not get status updates. 
> 
> In these cases, the user unfollows their fork of the project on CircleCI. This will trigger their jobs to run under the organization when they submit pull requests. Those users can optionally follow the source project if they wish to see the pipelines.

which as you said you did and the CI has started. Excellent work - now we would know what to tell the users in the future.

> 

Glad I could help! Thanks!