noamaanMulla-03/transformers-issues huggingface dataset:
add test for Jamba with new model jamba-tiny-dev
Author: yecohn
Pull request: yes

This PR uses the jamba-tiny-dev instead of jamba-tiny-random. It gives a better way to unittest Jamba. 

run-slow


Comments:
Hey! 🤗 Thanks for your contribution to the `transformers` library!

Before merging this pull request, slow tests CI should be triggered. To enable this:
- Add the `run-slow` label to the PR
- When your PR is ready for merge and all reviewers' comments have been addressed, push an empty commit with the command `[run-slow]` followed by a comma separated list of all the models to be tested, i.e. `[run_slow] model_to_test_1, model_to_test_2`
  - If the pull request affects a lot of models, put at most 10 models in the commit message
- A `transformers` maintainer will then approve the workflow to start the tests

(For maintainers) The documentation for slow tests CI on PRs is [here](https://www.notion.so/huggingface2/CI-for-pull-requests-8335bd9217d24d1e8ed1d3a016f39804).

Hey @yecohn, was this discussed in an issue or elsewhere? I'm wondering why the model change :)

Hey @LysandreJik  at ai21labs we trained  a tiny model for testing. It  offers a better tractability than the random-tiny model we uploaded.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33863). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
get error when running the chatglm3:  'GenerationConfig' object has no attribute '_eos_token_tensor'
Author: lmw0320
Pull request: no
Labels: name: bug

### System Info

platform == ubuntu 22.04
transformers == 4.32.2 
python == 3.10.12
<img width="776" alt="1721873757862" src="https://github.com/user-attachments/assets/abfc1fb6-a7e3-421c-a7b9-cb76d6a71df3">


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

git clone https://github.com/THUDM/ChatGLM3
cd ChatGLM3
pip install torch==2.2.2 torchvision==0.17.2 torchaudio==2.2.2 --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt
git clone https://huggingface.co/THUDM/chatglm3-6b
cd basic_demo
python3 cli_demo.py

### Expected behavior

I hope to run the command : python3 cli_demo.py successfully without any error

Comments:
@lmw0320 hey! ChatGLM uses code from the hub and is not maintained by us. There were changes to how special tokens are handled recently, and that affected ChatGLM's custom generation code. 

Downgrading transformers version to 4.42 will solve the issue, and you can report it at https://huggingface.co/THUDM/chatglm3-6b for the fix

cc @gante 

Downgrading transformers version to 4.39.2 slove this issue.
pip install transformers==4.39.2


It will also solved by using transformers up to version 4.41.2
`pip install transformers==4.41.2`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Closing then! 🤗
[i18n-ar] Translated file : `docs/source/ar/philosophy.md` into Arabic
Author: AhmedAlmaghz
Pull request: yes


## What does this PR do?
Translated the `docs/source/ar/philosophy.md` file of the documentation to `Arabic العربية`. I use transformers for my side-project and wanted to contribute. As a beginner, this seemed appropriate.

### Some questions:

- I can't access the pr documentation endpoint for `Arabic العربية`. Is there a missing step?
- I left all the other files alone after copying the English docs. Is this acceptable for future updates or should I add contents to the `_toctree.yml` only when translation is complete? Should I delete all other files for a cleaner review?
- In `Arabic`, people use both `Transformer` and `المحولات` to describe the model. However, as this is more of a product of HuggingFace I opted not to translate it. May you please let me know which you would prefer?

Thank you in advance for your review.

Part of #32435

### Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/) ?Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

 ### Who can review?
 @stevhliu , @MKhalusova , @abodacs, may you please review this PR?
 

Comments:
@abodacs

Next Review :
# CONCEPTUAL GUIDES

- [x]  Philosophy #33064
- [x]  Glossary #33038
- [ ]  What 🤗 Transformers can do #33073
- [ ]  How 🤗 Transformers solve tasks #33074
- [ ]  The Transformer model family #33047
- [ ]  Summary of the tokenizers #33078
- [ ]  Attention #33021
- [ ]  Padding and truncation #33050
- [ ]  BERTology #33024
- [ ]  Perplexity of fixed-length models #33063
- [ ]  Pipelines for webserver inference #33066
- [ ]  Model training anatomy #33045
- [ ] Getting the most out of LLMs #33043
[i18n-ar] Translated file : `docs/source/ar/task_summary.md` into Arabic
Author: AhmedAlmaghz
Pull request: yes


## What does this PR do?
Translated the `docs/source/ar/task_summary.md` file of the documentation to `Arabic العربية`. I use transformers for my side-project and wanted to contribute. As a beginner, this seemed appropriate.

### Some questions:

- I can't access the pr documentation endpoint for `Arabic العربية`. Is there a missing step?
- I left all the other files alone after copying the English docs. Is this acceptable for future updates or should I add contents to the `_toctree.yml` only when translation is complete? Should I delete all other files for a cleaner review?
- In `Arabic`, people use both `Transformer` and `المحولات` to describe the model. However, as this is more of a product of HuggingFace I opted not to translate it. May you please let me know which you would prefer?

Thank you in advance for your review.

Part of #32435

### Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/) ?Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

 ### Who can review?
 @stevhliu , @MKhalusova , @abodacs, may you please review this PR?
 

Comments:
@abodacs

Next Review :
# CONCEPTUAL GUIDES

- [x]  Philosophy #33064
- [x]  Glossary #33038
- [x]  What 🤗 Transformers can do #33073
- [ ]  How 🤗 Transformers solve tasks #33074
- [ ]  The Transformer model family #33047
- [ ]  Summary of the tokenizers #33078
- [ ]  Attention #33021
- [ ]  Padding and truncation #33050
- [ ]  BERTology #33024
- [ ]  Perplexity of fixed-length models #33063
- [ ]  Pipelines for webserver inference #33066
- [ ]  Model training anatomy #33045
- [ ] Getting the most out of LLMs #33043


Thank you @abodacs
[i18n-ar] Translated file : `docs/source/ar/tasks_explained.md` into Arabic
Author: AhmedAlmaghz
Pull request: yes


## What does this PR do?
Translated the `docs/source/ar/tasks_explained.md` file of the documentation to `Arabic العربية`. I use transformers for my side-project and wanted to contribute. As a beginner, this seemed appropriate.

### Some questions:

- I can't access the pr documentation endpoint for `Arabic العربية`. Is there a missing step?
- I left all the other files alone after copying the English docs. Is this acceptable for future updates or should I add contents to the `_toctree.yml` only when translation is complete? Should I delete all other files for a cleaner review?
- In `Arabic`, people use both `Transformer` and `المحولات` to describe the model. However, as this is more of a product of HuggingFace I opted not to translate it. May you please let me know which you would prefer?

Thank you in advance for your review.

Part of #32435

### Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/) ?Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

 ### Who can review?
 @stevhliu , @MKhalusova , @abodacs, may you please review this PR?
 

Comments:
اشكرك اخي عبدالله @abodacs  على جهودك الرائعة.

التالي :

# CONCEPTUAL GUIDES

- [x]  Philosophy #33064
- [x]  Glossary #33038
- [x]  What 🤗 Transformers can do #33073
- [x]  How 🤗 Transformers solve tasks #33074
- [ ]  The Transformer model family #33047
- [ ]  Summary of the tokenizers #33078
- [ ]  Attention #33021
- [ ]  Padding and truncation #33050
- [ ]  BERTology #33024
- [ ]  Perplexity of fixed-length models #33063
- [ ]  Pipelines for webserver inference #33066
- [ ]  Model training anatomy #33045
- [ ] Getting the most out of LLMs #33043
[i18n-ar] Translated file : `docs/source/ar/tokenizer_summary.md` into Arabic
Author: AhmedAlmaghz
Pull request: yes


## What does this PR do?
Translated the `docs/source/ar/tokenizer_summary.md` file of the documentation to `Arabic العربية`. I use transformers for my side-project and wanted to contribute. As a beginner, this seemed appropriate.

### Some questions:

- I can't access the pr documentation endpoint for `Arabic العربية`. Is there a missing step?
- I left all the other files alone after copying the English docs. Is this acceptable for future updates or should I add contents to the `_toctree.yml` only when translation is complete? Should I delete all other files for a cleaner review?
- In `Arabic`, people use both `Transformer` and `المحولات` to describe the model. However, as this is more of a product of HuggingFace I opted not to translate it. May you please let me know which you would prefer?

Thank you in advance for your review.

Part of #32435

### Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/) ?Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

 ### Who can review?
 @stevhliu , @MKhalusova , @abodacs, may you please review this PR?
 

Comments:
اشكرك اخي عبدالله @abodacs  على جهودك الرائعة.

التالي :

# CONCEPTUAL GUIDES

- [x]  Philosophy #33064
- [x]  Glossary #33038
- [x]  What 🤗 Transformers can do #33073
- [x]  How 🤗 Transformers solve tasks #33074
- [x]  The Transformer model family #33047
- [x]  Summary of the tokenizers #33078
- [ ]  Attention #33021
- [ ]  Padding and truncation #33050
- [ ]  BERTology #33024
- [ ]  Perplexity of fixed-length models #33063
- [ ]  Pipelines for webserver inference #33066
- [ ]  Model training anatomy #33045
- [ ] Getting the most out of LLMs #33043
[i18n-ar] Translated file : `docs/source/ar/model_summary.md` into Arabic
Author: AhmedAlmaghz
Pull request: yes


## What does this PR do?
Translated the `docs/source/ar/model_summary.md` file of the documentation to `Arabic العربية`. I use transformers for my side-project and wanted to contribute. As a beginner, this seemed appropriate.

### Some questions:

- I can't access the pr documentation endpoint for `Arabic العربية`. Is there a missing step?
- I left all the other files alone after copying the English docs. Is this acceptable for future updates or should I add contents to the `_toctree.yml` only when translation is complete? Should I delete all other files for a cleaner review?
- In `Arabic`, people use both `Transformer` and `المحولات` to describe the model. However, as this is more of a product of HuggingFace I opted not to translate it. May you please let me know which you would prefer?

Thank you in advance for your review.

Part of #32435

### Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/) ?Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

 ### Who can review?
 @stevhliu , @MKhalusova , @abodacs, may you please review this PR?
 

Comments:
بارك الله فيك اخي عبدالله @abodacs
[i18n-ar] Translated file : `docs/source/ar/attention.md` into Arabic
Author: AhmedAlmaghz
Pull request: yes


## What does this PR do?
Translated the `docs/source/ar/attention.md` file of the documentation to `Arabic العربية`. I use transformers for my side-project and wanted to contribute. As a beginner, this seemed appropriate.

### Some questions:

- I can't access the pr documentation endpoint for `Arabic العربية`. Is there a missing step?
- I left all the other files alone after copying the English docs. Is this acceptable for future updates or should I add contents to the `_toctree.yml` only when translation is complete? Should I delete all other files for a cleaner review?
- In `Arabic`, people use both `Transformer` and `المحولات` to describe the model. However, as this is more of a product of HuggingFace I opted not to translate it. May you please let me know which you would prefer?

Thank you in advance for your review.

Part of #32435

### Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/) ?Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

 ### Who can review?
 @stevhliu , @MKhalusova , @abodacs, may you please review this PR?
 

Comments:
Thank you @abodacs
[i18n-ar] Translated file : `docs/source/ar/pad_truncation.md` into Arabic
Author: AhmedAlmaghz
Pull request: yes


## What does this PR do?
Translated the `docs/source/ar/pad_truncation.md` file of the documentation to `Arabic العربية`. I use transformers for my side-project and wanted to contribute. As a beginner, this seemed appropriate.

### Some questions:

- I can't access the pr documentation endpoint for `Arabic العربية`. Is there a missing step?
- I left all the other files alone after copying the English docs. Is this acceptable for future updates or should I add contents to the `_toctree.yml` only when translation is complete? Should I delete all other files for a cleaner review?
- In `Arabic`, people use both `Transformer` and `المحولات` to describe the model. However, as this is more of a product of HuggingFace I opted not to translate it. May you please let me know which you would prefer?

Thank you in advance for your review.

Part of #32435

### Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/) ?Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

 ### Who can review?
 @stevhliu , @MKhalusova , @abodacs, may you please review this PR?
 

Comments:
Thank you @abodacs 

@AhmedAlmaghz 
 Adding the index causes conflicts and prevents the approval of the PR.
 I suggest adding/editing the index in a separate PR.
 
 What do you suggest @stevhliu ?
[i18n-ar] Translated file : `docs/source/ar/bertology.md` into Arabic
Author: AhmedAlmaghz
Pull request: yes


## What does this PR do?
Translated the `docs/source/ar/bertology.md` file of the documentation to `Arabic العربية`. I use transformers for my side-project and wanted to contribute. As a beginner, this seemed appropriate.

### Some questions:

- I can't access the pr documentation endpoint for `Arabic العربية`. Is there a missing step?
- I left all the other files alone after copying the English docs. Is this acceptable for future updates or should I add contents to the `_toctree.yml` only when translation is complete? Should I delete all other files for a cleaner review?
- In `Arabic`, people use both `Transformer` and `المحولات` to describe the model. However, as this is more of a product of HuggingFace I opted not to translate it. May you please let me know which you would prefer?

Thank you in advance for your review.

Part of #32435

### Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/) ?Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

 ### Who can review?
 @stevhliu , @MKhalusova , @abodacs, may you please review this PR?
 

Comments:
Thank you @abodacs
[i18n-ar] Translated file : `docs/source/ar/perplexity.md` into Arabic
Author: AhmedAlmaghz
Pull request: yes


## What does this PR do?
Translated the `docs/source/ar/perplexity.md` file of the documentation to `Arabic العربية`. I use transformers for my side-project and wanted to contribute. As a beginner, this seemed appropriate.

### Some questions:

- I can't access the pr documentation endpoint for `Arabic العربية`. Is there a missing step?
- I left all the other files alone after copying the English docs. Is this acceptable for future updates or should I add contents to the `_toctree.yml` only when translation is complete? Should I delete all other files for a cleaner review?
- In `Arabic`, people use both `Transformer` and `المحولات` to describe the model. However, as this is more of a product of HuggingFace I opted not to translate it. May you please let me know which you would prefer?

Thank you in advance for your review.

Part of #32435

### Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/) ?Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

 ### Who can review?
 @stevhliu , @MKhalusova , @abodacs, may you please review this PR?
 

Comments:
Thank you @abodacs