Add Phi-1 and Phi-1_5
Author: susnato
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #26110 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


[Colab Link](https://colab.research.google.com/drive/1rY-T36vqG9b-0IM3IJq741zBlcSxSit1?usp=sharing) to verify logits.

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

cc : @xenova 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerz and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Hey @susnato, thanks for opening your PR! Let us know if we can help in any way.

Also cc @gugarosa and @sgunasekar as you've been doing impressive maintenance work on your models :hugs: 

Hi @LysandreJik, this PR is ready for review.
Please refer to this [colab notebook](https://colab.research.google.com/drive/1rY-T36vqG9b-0IM3IJq741zBlcSxSit1?usp=sharing) to check the logits matching for `Phi-1` and `Phi-1.5`.

BTW the test - `test_left_padding_compatibility` seems to be failing here, but it is passing on my local machine. Do you know what could be causing this discrepancy?

 Nice, thanks a lot! Pinging @ArthurZucker and @Rocketknight1 to take a look :)

Will this PR support the training for phi models?

Hi @SinclairCoder, yes it will.

Hello @ArthurZucker, thanks for the suggestions and apologies for the delay.  

I have changed the code quite a bit according to the `persimmon`. For the `PhiAttention`, there is slight difference between the `_split_heads` methods of phi and persimmon so I could not just add `# Copied from` across the whole class, instead I have added that statement to the relevant methods seperately.
In addition to that please note that there are subtle differences in the `PhiDecoderLayer` and `PhiModel` too.

Please review it and let me know if you are ok with the changes or not. 

Hi @ArthurZucker ,  I have worked on the comments and now `PhiAttention` and `PhiModel` are same as their Persimmon counterparts. 

Please review it and let me know if this works with you. 

(also make sure the CI are green 🤗 )

Hello @ArthurZucker, I have pushed the changes and The CI is green now!

Regarding the BC compatibility issue - 
If it's ok with you, I would like to tackle that step by step after the PR is merged :smiley: .

1. Let's work together to make this PR merged.
2. After it is merged I will create a PR on the huggingface hub to upload both of the weights.
3. After that I will update the checkpoints, demo etc with separate PR just like we did for pop2piano. We can then tackle the compatibility issue in this step and update the config if needed.

Please let me know if this works with you :).

 

For sure, then we should use the same model_type as them, this way we won't have any problems and should not affect anyone! 

(The issue is taht other libraries like TGI use the model_type and otherwise we need to wait for them) 

Hi @ArthurZucker, I have updated the model_type as theirs(`"mixformer-sequential"`) and mentioned the error I am getting(after the updation) and temporary fix [here](https://github.com/huggingface/transformers/pull/26170#discussion_r1370124238).

I am also super down to have the correct model type but we know that's gonna have issue. Last time we had to revert the falcon PR 😓 We don't have the perfect solution yet, I can work on this before we actually push the checkpoints and the config if you are okay to wait? Otherwise we can use their model type for now and later on once I find the correct way to approach this we'll update! 😉 

> I am also super down to have the correct model type but we know that's gonna have issue. Last time we had to revert the falcon PR 😓 We don't have the perfect solution yet, I can work on this before we actually push the checkpoints and the config if you are okay to wait?

If you think that's relatively better solution, then I will wait. 

But then could you please explain me the whole process a little bit? I mean, will you first work towards changing the `model_type` to "phi" on the Hub then this PR and then actually push the checkpoints and the config?

Also I have a question - after the PR is merged, we will need to delete the modelling files on the HuggingFace Hub right? So that everyone loads the model from the library only?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26170). All of your documentation changes will be reflected on that endpoint.

when importing phi with transformers == 4.34.1  `from transformers import PhiModel`
[](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26170/en/model_doc/phi)

I get an error: ImportError: cannot import name 'PhiForCausalLM'

Has the updates been made yet? Just seeing if this is expected. 


Hey @susnato, we'll probably merge this PR, then open a PR to the repo on the hub with the new checkpoints and a deletion of the code, making sure checking out this revision works, then merging once the release happens (next week). I'll let @Rocketknight1 help you with this as he went thought that for Falcon! 

Regarding this PR, let's merge it but with the correct model_type (so the one they have in their config)  for now. I didn't have time to dive into fixing and will be off for a week so will tackle after that! 

Hey @ArthurZucker, I tried to change to `mixformer-sequential` for `MODEL_NAMES_MAPPING` and `CONFIG_MAPPING_NAMES` but it seems that I need to change the whole directory from `phi` to `mixformer-sequential` in order to make it work.
 

Otherwise I believe that we can also add  `("mixformer-sequential", "phi")` to `SPECIAL_MODEL_TYPE_TO_MODULE_NAME` so that every `mixformer-sequential` will get converted to `phi`.

WDYT?
 

Mmm yeah something like `HUB_MODEL_TYPE_TO_TRANSFORMERS_MODEL_TYPE` will have to be created, but it's a bit more involved we need to test and make sure we handle edge case

I'll try to do this today 👍🏻 

Okk thanks!

Just replying here to say that yes, I'm available to help when you're ready to start porting checkpoints!

Hello @Rocketknight1, thanks for the reply!

I have already ported the checkpoints and saved here - [phi-1](https://huggingface.co/susnato/phi-1_dev) and [phi-1.5](https://huggingface.co/susnato/phi-1_5_dev)

The main [issue](https://github.com/huggingface/transformers/pull/26170#pullrequestreview-1692913227) that @ArthurZucker was talking about was the difference between the `model_type`, the [online config file, on the hub](https://huggingface.co/microsoft/phi-1/blob/371fd519ab41b5bc2e96838936c2349dc49656db/config.json#L14) has model_type as `mixformer-sequential` but here we have it as `phi`. This won't matter if we change the model_type to phi on the Hub.
Also I believe that we will need to delete the modelling files on the HuggingFace Hub right?

So, how should I approach this? should I first open a PR to transfer the weights and config to the original repo on Hub or this PR needs to be merged first then the weights transfer? WDYT? 


@susnato This PR will need to be merged first, because otherwise the updated repos won't work for anyone else.

After that, the way we did it was to make a PR to each checkpoint repo, and do the following:

1) Copy the `configuration_phi.py` and `modeling_phi.py` files from this PR into the repo, overwriting the old ones.
2) Ensure the files don't use relative imports (e.g. replace `from ..modeling_utils import` with `from transformers.modeling_utils import`
3) Set the `architectures` and `auto_map` keys in `config.json` to point to the new classes.
4) Set the `model_type` to the new type (e.g. `phi` in this case)

This way, users who have updated to the latest version of `transformers` can use `trust_remote_code=False` and get the library code, but users who haven't can continue with their old workflows.

Hi @Rocketknight1, thanks a lot for the explanation!

So just to be clear - we will merge this PR with `model_type = "phi"` right? Or we need to use `mixformer-sequential`?

@susnato we should use "phi"

@Rocketknight1, Okk. Then lets try to work towards merging this first.

If you don't mind could you please review this? :sweat_smile:  

@susnato will do! Please do the rebase anyway though, lol. I think we'll get lucky, and if we don't then it's better to find out here then after we merge!

Hello @Rocketknight1, I have rebased and did necessary changes.

Please note that I have also added an [dropout layer](https://github.com/susnato/transformers/blob/26cceddd5504d5f58d899469483a0245f082f0ee/src/transformers/models/phi/modeling_phi.py#L610) after the `input_embeds` which was missing in the previous commits. 

Got it! Pinging either @arthurzucker or @amyeroberts for core maintainer final review. I checked with the TGI team and they said it would be fine to do the port and change the model type to `phi`.

Hi @amyeroberts, I have reworked the model doc page. Let me know if this is okay to you. 

Looks like the CI error is unrelated to this PR. Should I rebase and push again? @amyeroberts 

@susnato Yes please! Sorry - there were a few unexpected error because of new packages releases and incompatibilities. These should now be resolved on main.

Hi, @amyeroberts I just rebased but the failing test seems to be unrelated. 


@susnato Indeed. Really sorry about this, we're trying to resolve now. It seems there's some flakiness in accelerate tests - which means even if we _think_ we've fixed them another rears its head. I'll let you know once the latest patch is merged. Fingers crossed that will be the last rebase you'll have to do. 

No problem @amyeroberts, just ping me when it's patched and I will rebase then. 

OK, third time lucky. Apologies for the delay in getting this merged and thank you for your patience. A fix which has been run across the full test suite has been merged into main. If you rebase this _should_ resolve the issue 

Hmm it failed on another test :( but it was ok for the CLVP PR,  let me make some minor doc changes and push one more time.

Hi @amyeroberts, the CI is green now! 

@susnato Amazing! Thanks for your work adding this and your patience whilst we sorted out the CI

Hi @susnato, congrats on the PR! Let me know if you're willing to tackle opening PRs to the model repos as well. I mentioned the recommended approach [here](https://github.com/huggingface/transformers/pull/26170#issuecomment-1787585303).

Hi @Rocketknight1, yes I will open PR to both of the repos on the Hub this week.

Amazing, thank you!

Hi @Rocketknight1, sorry I could not open the PRs this week, I will try to complete them ASAP.

@susnato Don't stress it, there's no rush! We (and the whole community) really appreciate the work you're putting in on this one

Hi @Rocketknight1, I have created both the PRs - [phi-1](https://huggingface.co/microsoft/phi-1/discussions/5) and [phi-1.5](https://huggingface.co/microsoft/phi-1_5/discussions/62).

Could you please check them and let me know if anything needs to be changed? 

Hi all,

It seems that the current code for Phi-1.5 does not support gradient checkpointing. Could you support this?

Hi @liziniu, gradient checkpointing is already supported by `Phi`.

```python
from transformers import PhiForCausalLM

model = PhiForCausalLM.from_pretrained("susnato/phi-1_5_dev") # the weights are currently being transferred to microsoft/phi-1_5
print(model.model.gradient_checkpointing) # False

# enable Gradient Checkpointing
model.gradient_checkpointing_enable()

print(model.model.gradient_checkpointing) # this should be True now.
``` 

If you want to freeze some internal layers alongside then please add this kwarg - 
```python
model.gradient_checkpointing_enable(gradient_checkpointing_kwargs={"use_reentrant": False})
``` 

Hi @susnato, aiming to review today!

Hello folks

I've encountered an issue while attempting to load my fine-tuned model after the recent update. The error message I received is as follows:

"OSError: microsoft/phi-1_5 does not seem to contain a file named configuration_mixformer_sequential.py. Please check 'https://huggingface.co/microsoft/phi-1_5/main' for available files."

As someone relatively new to this, can anyone help me figure out how to tweak my code so I can load my fine-tuned model properly?
```
model = AutoModelForCausalLM.from_pretrained("nulltella/phi-1_5-finetuned-model", trust_remote_code=True, torch_dtype=torch.float32)
tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-1_5", trust_remote_code=True)
```
Thanxxx !
#Help

Hi @mohamedjihedriahi, as far as I can tell from the error message, your code is trying to find `configuration_mixformer_sequential.py` but that file was deleted in this [commit](https://huggingface.co/microsoft/phi-1_5/commit/8e9ebfb9bfe4da87d5c705d097fe884dc8e62d6f). So you are getting this error.


**Short Time Fix**
For a quick fix, you can try to convert your `nulltella/phi-1_5-finetuned-model` to have the same `config.json` as `microsoft/phi-1_5` according to [this commit](https://huggingface.co/microsoft/phi-1_5/commit/271c3397ab4e1f8f4e49868b1e8ba0be95363c88).

**Long Time Fix (Recommended)**
Since `phi` is now added to `transformers`, I would suggest you - 
1. upgrade to the latest `transformers`.
2. convert the previous weights according to the library model's weight structure. (this would require you to rearrange the keys of the weights).

This way, you won't have to care about any new changes to the model `phi` in the library. 


Let me know if you need any assistance/help in doing this.