Generate using exported model and enable gemma2-2b in ExecuTorch
Author: guangy10
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Adding `generate` support for exported model.
Adding `gemma2-2b` to `ExecuTorch` with tests.
Adding an integration test for `gemma-2b` that we've enabled already. 

### Additional Test in `ExecuTorch`
Running `gemma2-2b` E2E:
```
cmake-out/examples/models/llama2/llama_main --tokenizer_path=tokenizer_gemma2.bin --model_path=gemma2.pte --prompt="My name is"
I 00:00:00.001356 executorch:cpuinfo_utils.cpp:62] Reading file /sys/devices/soc0/image_version
I 00:00:00.001425 executorch:cpuinfo_utils.cpp:78] Failed to open midr file /sys/devices/soc0/image_version
I 00:00:00.001431 executorch:cpuinfo_utils.cpp:158] Number of efficient cores 4
I 00:00:00.001434 executorch:main.cpp:65] Resetting threadpool with num threads = 6
I 00:00:00.005564 executorch:runner.cpp:65] Creating LLaMa runner: model_path=gemma2.pte, tokenizer_path=tokenizer_gemma2.bin
E 00:00:03.701808 executorch:tiktoken.cpp:79] invalid tiktoken line:
I 00:00:03.701845 executorch:runner.cpp:88] Failed to load tokenizer_gemma2.bin as a Tiktoken artifact, trying BPE tokenizer
I 00:00:03.767485 executorch:runner.cpp:94] Reading metadata from model
I 00:00:03.767512 executorch:runner.cpp:119] Metadata: get_vocab_size = 256000
I 00:00:03.767515 executorch:runner.cpp:119] Metadata: get_bos_id = 2
I 00:00:03.767517 executorch:runner.cpp:117] Methond use_sdpa_with_kv_cache not found, using the default value 0
I 00:00:03.767518 executorch:runner.cpp:119] Metadata: use_sdpa_with_kv_cache = 0
I 00:00:03.767520 executorch:runner.cpp:119] Metadata: get_n_eos = 1
I 00:00:03.767521 executorch:runner.cpp:117] Methond append_eos_to_prompt not found, using the default value 0
I 00:00:03.767522 executorch:runner.cpp:119] Metadata: append_eos_to_prompt = 0
I 00:00:03.767524 executorch:runner.cpp:119] Metadata: get_max_seq_len = 123
I 00:00:03.767525 executorch:runner.cpp:117] Methond enable_dynamic_shape not found, using the default value 0
I 00:00:03.767527 executorch:runner.cpp:119] Metadata: enable_dynamic_shape = 0
I 00:00:03.767529 executorch:runner.cpp:119] Metadata: use_kv_cache = 1
I 00:00:03.767575 executorch:runner.cpp:119] Metadata: get_n_bos = 1
I 00:00:03.767604 executorch:runner.cpp:167] RSS after loading model: 0.000000 MiB (0 if unsupported)
I 00:00:04.408489 executorch:runner.cpp:234] RSS after prompt prefill: 0.000000 MiB (0 if unsupported)
My name is Lale and I love to play with my dolls. I started to play with dolls at the age of two years old. My favorite activity is dancing. I would like to help people. I would like to travel to Spain. I like to be a vet. I love to help people. I would like to travel. I would like to be. I love to travel. I would like to be. I like to travel. I would like to be. I love to travel. I would like to be. I love to travel. I would like to be. I love to travel
I 00:00:23.188418 executorch:runner.cpp:246] RSS after finishing text generation: 0.000000 MiB (0 if unsupported)
PyTorchObserver {"prompt_tokens":4,"generated_tokens":118,"model_load_start_ms":1727310065089,"model_load_end_ms":1727310068851,"inference_start_ms":1727310068851,"inference_end_ms":1727310088272,"prompt_eval_end_ms":1727310069492,"first_token_ms":1727310069492,"aggregate_sampling_time_ms":180,"SCALING_FACTOR_UNITS_PER_SECOND":1000}
I 00:00:23.188436 executorch:stats.h:84] 	Prompt Tokens: 4    Generated Tokens: 118
I 00:00:23.188438 executorch:stats.h:90] 	Model Load Time:		3.762000 (seconds)
I 00:00:23.188440 executorch:stats.h:100] 	Total inference time:		19.421000 (seconds)		 Rate: 	6.075897 (tokens/second)
I 00:00:23.188442 executorch:stats.h:108] 		Prompt evaluation:	0.641000 (seconds)		 Rate: 	6.240250 (tokens/second)
I 00:00:23.188444 executorch:stats.h:119] 		Generated 118 tokens:	18.780000 (seconds)		 Rate: 	6.283280 (tokens/second)
I 00:00:23.188446 executorch:stats.h:127] 	Time to first generated token:	0.641000 (seconds)
I 00:00:23.188480 executorch:stats.h:134] 	Sampling time over 122 tokens:	0.180000 (seconds)
```


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. #33709
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker
@gante
@amyeroberts
@qubvel 


Comments:
Why am I start getting `fetch_tests - Unauthorized` error on all the PRs I'm creating?

This seems to be happening more often, @ydshieh would you know what might be happening? Otherwise I'll reach out to the CircleCI team, this is hindering work on the repo

I've asked the CircleCI team @guangy10, very sorry for the inconvenience.

Might be related to 

> Allow CI could be run on private forked repositories (e.g. new model additions) (#33594)

But not happening on all external contributor's PRs. Strange

Could you first try ..?

https://support.circleci.com/hc/en-us/articles/360048210711-How-to-Refresh-User-Permissions





Another thing to check

[If you're following the fork instead of the upstream repo](https://support.circleci.com/hc/en-us/articles/360008097173-Troubleshooting-why-pull-requests-are-not-triggering-jobs-on-my-organization)

```
A user who submits a pull request to your repository from a fork, but no pipeline is triggered with the pull request. This can happen when the user is following the project fork on their personal account rather than the project itself on CircleCI.

This will cause the jobs to trigger under the user's personal account. If the user is following a fork of the repository on CircleCI, we will only build on that fork and not the parent, so the parent’s PR will not get status updates. 

In these cases, the user unfollows their fork of the project on CircleCI. This will trigger their jobs to run under the organization when they submit pull requests. Those users can optionally follow the source project if they wish to see the pipelines.
```

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33707). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Could you first try ..?
> 
> https://support.circleci.com/hc/en-us/articles/360048210711-How-to-Refresh-User-Permissions

Weird. Used to work fine on my old PRs. Tried "Refresh Permission". Let's see if it can be unblocked. 

@ydshieh  It's getting worse after re-fresh the permissions. check_circleci_user starts failing.

> Another thing to check
> 
> [If you're following the fork instead of the upstream repo](https://support.circleci.com/hc/en-us/articles/360008097173-Troubleshooting-why-pull-requests-are-not-triggering-jobs-on-my-organization)
> 
> ```
> A user who submits a pull request to your repository from a fork, but no pipeline is triggered with the pull request. This can happen when the user is following the project fork on their personal account rather than the project itself on CircleCI.
> 
> This will cause the jobs to trigger under the user's personal account. If the user is following a fork of the repository on CircleCI, we will only build on that fork and not the parent, so the parent’s PR will not get status updates. 
> 
> In these cases, the user unfollows their fork of the project on CircleCI. This will trigger their jobs to run under the organization when they submit pull requests. Those users can optionally follow the source project if they wish to see the pipelines.
> ```

"In these cases, the user unfollows their fork of the project on CircleCI. " I have no idea how to unfollow my fork on CircleCI. I don't even see if I'm following transformers on CircleCI. I created a CircleCI account with exact email and linked to my github, and I can't see any project I'm following..

https://support.circleci.com/hc/en-us/articles/360008097173-Troubleshooting-why-pull-requests-are-not-triggering-jobs-on-my-organization
> so please be sure "Build forked pull requests" is enabled in Project Settings > Advanced

@ydshieh Their Wiki is so bad. I can't find this setting from anywhere.

Hmm, weird CircleCI issue ...

Could you check a last time:

top-left (like in the image): organization : switch to your own org, follow `transformers` there.
Optionally: Then switch to `huggingface` (if you can find it) and follow `transformers` there

If still not working, I could try to push a commit.

<img width="1258" alt="Screenshot 2024-09-28 115104" src="https://github.com/user-attachments/assets/a2e5cc56-cb0a-44e3-9c20-6c5abaa3a0c3">


I push a commit to trigger the CI jobs. It runs now. Got to say I have no idea what is wrong on the CircleCI side with the permission issue as other external contributors' PRs don't have it.

Hi @ydshieh thank you so much for debugging this issue together with me. Per the message from CircleCI support team, it seems like the `TRANSFORMERS_CONTEXT` in the `.circleci/config.yml` is causing the permission issue, and I was suggested to remove it from the `config.yml`. It seems like it's newly added in #33594 last week. Still I'm not sure why it only affects my PRs but works for other users, but hope this message can give you more pointer to help resolve this issue on my fork and PRs. Thank you.


![Screenshot 2024-09-30 at 11 42 35 AM](https://github.com/user-attachments/assets/4fd40324-e962-40a9-aa50-d4e3cd10eac8)



Since @ydshieh pushed a commit to trigger the CI and all CIs are green. Can I get a review on this PR? @amyeroberts @qubvel 

Hi @guangy10 Thank you for the message and contacting CircleCI team on your side too. I also think it is from #33594 from the beginning of seeing this issue. But as you mentioned, only a few (external) contributors face this issue, and the answer from CircleCI attached above is somehow confusing (i.e. it doesn't explain what **actually** causes it).

I will create a new personal github account and see if I can reproduce and come up with a solution.



@guangy10 Could you go https://app.circleci.com/home/ and see what you get there? 🙏 

And https://app.circleci.com/projects/project-dashboard/github/guangy10/

Well I am able to reproduce now 

https://github.com/huggingface/transformers/pull/33850

Opened a PR #33866 for this CircleCI issue

> Opened a PR #33866 for this CircleCI issue

@ydshieh Thanks a lot. Really appreciate the quick fix! Let me rebase on top of the fix

Hey! 🤗 Thanks for your contribution to the `transformers` library!

Before merging this pull request, slow tests CI should be triggered. To enable this:
- Add the `run-slow` label to the PR
- When your PR is ready for merge and all reviewers' comments have been addressed, push an empty commit with the command `[run-slow]` followed by a comma separated list of all the models to be tested, i.e. `[run_slow] model_to_test_1, model_to_test_2`
  - If the pull request affects a lot of models, put at most 10 models in the commit message
- A `transformers` maintainer will then approve the workflow to start the tests

(For maintainers) The documentation for slow tests CI on PRs is [here](https://www.notion.so/huggingface2/CI-for-pull-requests-8335bd9217d24d1e8ed1d3a016f39804).

> * Add the `run-slow` label to the PR

@ydshieh I don't see there is a way for me to add `run-slow` label to the PR. Is there one?

Are you able to see what is shown below?

<img width="991" alt="Screenshot 2024-10-01 200227" src="https://github.com/user-attachments/assets/9abebb94-45da-4e3a-a9a1-d42f63ce96c9">


> Are you able to see what is shown below?
> 
> <img alt="Screenshot 2024-10-01 200227" width="991" src="https://private-user-images.githubusercontent.com/2521628/372594758-9abebb94-45da-4e3a-a9a1-d42f63ce96c9.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mjc4MDY4NjUsIm5iZiI6MTcyNzgwNjU2NSwicGF0aCI6Ii8yNTIxNjI4LzM3MjU5NDc1OC05YWJlYmI5NC00NWRhLTRlM2EtYTlhMS1kNDJmNjNjZTk2YzkucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MTAwMSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDEwMDFUMTgxNjA1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9OGQ2YmNiY2M0OGFjZDMxNDRkNDE3ODc3NWYzNjQ4MjQyNTRjZjc1NTA4MjJkY2NiZWIxZDkyY2NkODEyZDhmMyZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QifQ.9fIh9YuMAi5q5TFtbUDUNBU3TS0Ji22ybkKnbouN-AI">

No, that's why I'm asking if the message is asking me to do so or the repo maintainers. Also wondering if we have a something like `pytorchbot` that can do labeling for any user w/o having write permission to the repo.
 
![Screenshot 2024-10-01 at 11 29 21 AM](https://github.com/user-attachments/assets/ab8d7112-5f2d-4dc6-a31e-f81e3a50b754)


Thanks for the feedback. We will improve the stuffs. For now I just added it manually here.

@ArthurZucker @qubvel could you help review for this PR? Once it's merged we can add such integration tests for all executorch-compatible models, not only test the exportability but also the generate.

Don't forget to push an (empty) commit with message: `[run_slow] gemma, gemma2`, thank you

Comments addressed.
Does it require a 2nd reviewer in order to merge?

Nope, just waiting for the CIs right now! 

Debug the `gemma2` export issue and update the required `torch` version for it.