Add EXAONE
Author: lgai-exaone
Pull request: yes
Labels: name: New model

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Add [EXAONE](https://github.com/LG-AI-EXAONE/EXAONE-3.0) model released by LG AI Research.

Test code and documentation are currently in progress.

Please refer to the corresponding issue: https://github.com/huggingface/transformers/issues/34651


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@ArthurZucker 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Please review this PR. @ArthurZucker @itazap 

Hey! Of course 🤗 sorry for being late we were on a company wide offsite!

I think it has some probability that purely build based on transformer? Maybe asking training script could be one way

Hey! Thanks so much for your kind words and for reaching out! 🚀 We're thrilled to hear your interest in contributing to EXAONE and exploring its integration possibilities.

To address your question:

**1. Key Differences from LLaMA**

EXAONE differs from LLaMA in several foundational ways:
- **Tokenizer:** While LLaMA 2 uses LlamaTokenizer and LLaMA 3 adopts Tiktoken, EXAONE uses GPT2Tokenizer as its default tokenizer.
- **Impact of Tokenizer on Performance:** A common issue occurs when users apply LLaMA's tokenizer after converting EXAONE into a LLaMA-style implementation. This often leads to significant accuracy degradation, particularly due to <bos> token handling in LLaMA's tokenizer, which doesn't align with EXAONE's design.

While adding a conversion script might address token compatibility, it doesn't fully resolve the potential issues stemming from deeper differences in model architecture and training methodology.

**2. Why EXAONE Modeling Should Remain Independent**

EXAONE was trained entirely from scratch and has no direct relation to LLaMA, though we’ve noticed a tendency for people to misunderstand this point. As EXAONE continues to evolve, its implementation may diverge even further from LLaMA as we experiment with novel techniques and enhancements. Relying on LLaMA's modeling could limit our flexibility to innovate and adapt.
Furthermore, as a bilingual model with the unique characteristic of supporting the Korean language, unlike English-centric models, managing EXAONE separately becomes an even more critical issue.
Our ultimate goal is to open-source EXAONE in a way that maximizes clarity, usability, and scalability—preserving the model's unique characteristics while empowering contributors like you to make the most of its capabilities. 

We’re excited to collaborate with you and would love to hear your thoughts or ideas on how we can move forward.

Please consider it positively. We will also contribute a lot. 🤗

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34652). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hey! Sorry for the late answer. 
I will answer as I have always done, this goes against the philosophy of transformers! 
We are trying our best to set good standards, and by making sure you model is converted to the llama format, you will make sure it's available day-0 on all the other framework, making it available for the entire community at 0 software cost!

Super happy to see a new model, and I think that good results mean your team did a great job a training it 🚀 

In transformers, we don't change past modeling code to adapt new checkpoints, for us, a different codepath means a new architecture! 

The tokenizer should simply use `PreTrainedTokenizerFast` , this can be changed in the `config.json`! 

Hey! Thank you for your response.
 
We understand and respect the transformers library's philosophy of maintaining consistent architecture standards.
 
However, as we mentioned before, we have decided to maintain a separate modeling from the LLaMA's for future development needs, while we acknowledge the benefits of cross-framework compatibility.
 
We look forward to exploring ways to integrate EXAONE's distinct architectural features with the transformers library in a standardized way and to continuing our constructive discussions in the future.
 
We truly appreciate your time in reviewing our model and your valuable contributions to the open-source community.

Please accept this new architecture or whatever the issues is...I've tested out the exaone models and it beats out qwen's comparably sized models.  The 7.8b model was the only model of comparable size to achieve a perfect score on my custom benchmark, which I can provide details regarding if you want.  It's specifically geared towards typical RAG questions - e.g. needle in haystack, chain reasoning (e.g. how many albums has XYZ artist released when the LLM has to count from multiple contexts), etc.

I'm having trouble understanding the issue with forcing them to use Llama's architecture.  Sorry, this isn't my profession.

EDIT - I'm referring to the exaone 3.5 models.

Hey! You can already use the model by simply converting layer names! The same drama happened with QWEN 1 model and the authors saw that it was more beneficial to accept llama architecture. 
The key idea is that this model IS a llama model, with absolutely 0 differences in terms of architecture. There is 0 motivation to not use an architecture supported by more than 20 softwares (`transformers`, `TGI`, `vllm`, `peft`, `llama.cpp`, `ollama`, `mlx`, etc) when with absolutely 0 cost the model can be used. 

I am super happy to promote LG's work, as I believe they pulled off something super nice by training this model and pushing it's performances! 

Regarding: 
> we have decided to maintain a separate modeling from the LLaMA's for future development needs, while we acknowledge the benefits of cross-framework compatibility.

This is not something that can happen in transformers as for us, 1 architecture == 1 codepath, meaning that we never go back and patch a modeling code, if there is a newer version, we just add a new model! This is how we have been doing for the past 4 years! 🤗