noamaanMulla-03/transformers-issues huggingface dataset:
4.34 cause baichuan tokenizer break
Author: nrailg
Pull request: no

### System Info

transformers==4.34

### Who can help?

@ArthurZucker and @younesbelkada

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Use AutoTokenizer to load Baichuan 7B https://huggingface.co/baichuan-inc/Baichuan-7B/blob/main/tokenization_baichuan.py

```
  File "/home/nrwu/work/stanford_alpaca/app/sug-dpo/sft.py", line 94, in <module>
    tokenizer = AutoTokenizer.from_pretrained(
  File "/home/pretrainx/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 738, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/home/pretrainx/conda/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2045, in from_pretrained
    return cls._from_pretrained(
  File "/home/pretrainx/conda/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2256, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/nrwu/work/hf-home/modules/transformers_modules/Baichuan-7B/tokenization_baichuan.py", line 75, in __init__
    super().__init__(
  File "/home/pretrainx/conda/lib/python3.10/site-packages/transformers/tokenization_utils.py", line 366, in __init__
    self._add_tokens(self.all_special_tokens_extended, special_tokens=True)
  File "/home/pretrainx/conda/lib/python3.10/site-packages/transformers/tokenization_utils.py", line 462, in _add_tokens
    current_vocab = self.get_vocab().copy()
  File "/home/nrwu/work/hf-home/modules/transformers_modules/Baichuan-7B/tokenization_baichuan.py", line 109, in get_vocab
    vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)}
  File "/home/nrwu/work/hf-home/modules/transformers_modules/Baichuan-7B/tokenization_baichuan.py", line 105, in vocab_size
    return self.sp_model.get_piece_size()
AttributeError: 'BaiChuanTokenizer' object has no attribute 'sp_model'
```

### Expected behavior

there should be no exception.

Comments:
upgrade transformers to 4.33.2





> upgrade transformers to 4.33.2

Thanks, I'll try it.

It's "downgrade", precisely.

Hey! You should open an issue on the repo as this is a `remote` code, which we have no real control over. The fix is fairly straightforward, but should be done by the authors to make sure they have the expected behaviour.  

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Unable to achieve the same accuracy between Trainer API and tensorflow models
Author: KerenzaDoxolodeo
Pull request: no

### System Info


- `transformers` version: 4.33.0
- Platform: Linux-6.1.42+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: 0.22.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.0.0 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.2 (gpu)
- Jax version: 0.4.13
- JaxLib version: 0.4.13
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I run xlm-roberta in three implementations:

1) Using TFAutoModelForSequenceClassification
2) Using TFAutoModel, with the classification layer as faithful as possible to huggingface's implementation.

Code : https://www.kaggle.com/code/realdeo/keras-code/settings?scriptVersionId=145530298

3) Using TrainerAPI

Code : https://www.kaggle.com/code/realdeo/fork-of-notebookcb67cb4ef2/notebook?scriptVersionId=145540775

### Expected behavior

I expect the code to have roughly the same accuracy. What happens here is the Trainer API successfully trained after 1 epoch while the tensorflow implementation stuck at predicting the same label. 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @KerenzaDoxolodeo, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix `JumanppTokenizer` to deal with half-width spaces properly
Author: hkiyomaru
Pull request: yes

# What does this PR do?

This PR addresses the issue with the `JumanppTokenizer` that incorrectly handled half-width spaces by replacing them with full-width spaces. This problem originated from a bug in the dependent library, `rhoknp`. This PR resolves this bug by updating `rhoknp` to the latest version.

Before this PR:

```python
>>> tokenizer = JumanppTokenizer()
>>> tokenizer.tokenize("foo\u2009bar")  # \u2009: half-width space
>>> ["foo", "\u3000", "bar"]  # \u3000: full-width space (bug)
```

With this PR:

```python
# With this PR
>>> tokenizer = JumanppTokenizer()
>>> tokenizer.tokenize("foo\u2009bar")  # \u2009: half-width space
>>> ["foo", "\u2009", "bar"]
```

This PR also includes updates to the test cases for `JumanppTokenizer` to ensure its proper functionality.

It's important to note that this PR changes the behavior of `JumanppTokenizer`. Even though the change corrects a bug, some existing models might be built upon the earlier behavior and could be impacted.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@ArthurZucker 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26524). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix remaining issues in beam score calculation
Author: VsonicV
Pull request: yes

# What does this PR do?
This PR further fixes the remaining issues in beam score calculation following #27351 .
More specifically:
1) When adding new hypothesis, the `hyp` in `process` does not include the next generated token on which the current beam score is calculated, but the `hyp` in `finalize` includes all the generated tokens so far. This inconsistency is resolved by changing the `add` function of `BeamHypotheses`. Now we directly pass the current length of the generated tokens to `add`.
2) When calculating best possible beam score in `is_done` function of `BeamHypotheses`, we are directly using `max_length` without deducting `decoder_prompt_len`. It is fixed now.
3) Updated the testing expectation accordingly.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?


## Who can review?
@gante 


Comments:
> Good catch, thanks for fixing!
> 
> BTW, run `RUN_SLOW=1 py.test tests/models/vision_encoder_decoder/test_modeling_vision_encoder_decoder.py::ViT2GPT2ModelIntegrationTest::test_inference_coco_en` -- this test may need an update in its value

@gante I have updated the expectation value for this test. I have also incorporated all your suggestions. Ready to go!

@gante I have also updated the usage of `cur_len` in this Pytorch version following your suggestions in #27814 , now it represents the length of the entire sequence including the decoder prompt, which is consistent with the remaining codebase.

All suggested changes are incorporated. Ready to go! @gante @ArthurZucker
Can't get add_generation_prompt to work correctly in apply_chat_template
Author: odellus
Pull request: no

I'm having trouble getting the `add_generation_prompt` feature working with `tokenizer.apply_chat_template`. I'm working with stablelm-zephyr-3b right now. I raised an issue on their HF model page, but I don't think the problem is with their chat template. Their chat template looks correct.

https://huggingface.co/stabilityai/stablelm-zephyr-3b/discussions/9

Discussion reproduced here so you don't have to click through:

Not able to get `tokenizer.apply_chat_template` to append the generation prompt for stablelm-zephyr-3b

```python
print(tokenizer.chat_template)
"{% for message in messages %}\n{% if message['role'] == 'user' %}\n{{ '
\n' + message['content'] + eos_token }}\n{% elif message['role'] == 'system' %}\n{{ '
\n' + message['content'] + eos_token }}\n{% elif message['role'] == 'assistant' %}\n{{ '
\n'  + message['content'] + eos_token }}\n{% endif %}\n{% if loop.last and add_generation_prompt %}\n{{ '
' }}\n{% endif %}\n{% endfor %}"

chat = [{'role': 'system', 'content': 'You are an excellent C++ programmer'}, {'role': 'user', 'content': 'Write a program to compute pairwise distances between atoms in a PDB file'}]

tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)
'
\nYou are an excellent C++ programmer
\n
\nWrite a program to compute pairwise distances between atoms in a PDB file
\n'

tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=False)
'
\nYou are an excellent C++ programmer
\n
\nWrite a program to compute pairwise distances between atoms in a PDB file
\n'
```

Could this be an issue with tokenizer module? The chat template looks right.

Comments:
cc @Rocketknight1 if you have time to look at this! 

I'm using `transformers==4.34.0`

Upgrading to 4.35.2 fixed this issue for me. Closing.
Has anyone used run_language_modelling.py to train a gpt 2 from scratch? 
Author: nikkon3
Pull request: no
Labels: name: Discussion

name: wontfix

name: Ex: LM (Pretraining)

I have read this post https://huggingface.co/blog/how-to-train and i would like to do the train from scratch, for a gpt2 type model. 
One question also, that i have, is the special tokens that he uses in that post for tokenizer will be the same and for a tokenizer that i will use for a gpt2 model?

Comments:
Hi @nikkon3, the special tokens for `gpt2` are automatically set when you import `GPT2Tokenizer`.

The code below shows that `'
'` is the special token used for BOS (beginning of sequence), EOS (end of sequence), and UNK (unknown - out of vocabulary).

```
from transformers import GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
tokenizer.special_tokens_map

#{'bos_token': '
',
# 'eos_token': '
',
# 'unk_token': '
'}
```

Hope this helps you out!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


you can use run_clm.py to train gpt2 from scratch in transformers 3.5.1
T5 automatic creation of the `encoder_attention_mask`
Author: pietrolesci
Pull request: no

Hi there,

I am going through the T5 implementation to try to replicate it and noticed that the two code blocks reported below do the same thing when creating the `encoder_attention_mask`. Is this wanted?

https://github.com/huggingface/transformers/blob/037fb7d0e1086146612a716ef914305160134e9c/src/transformers/models/t5/modeling_t5.py#L1029-L1033

https://github.com/huggingface/transformers/blob/037fb7d0e1086146612a716ef914305160134e9c/src/transformers/models/t5/modeling_t5.py#L1045-L1049

Comments:
Hi @pietrolesci, thanks for raising this issue.

Indeed, it seems the second mask creation is redundant. It was probably just a copy-pasta that got added along the way. Would you like to open a PR to fix it? This way you get the github contribution for having eagle eyes 🦅 👀  

Hi @amyeroberts, thanks for your swift reply and for confirming. Yes, I will open a PR later today :)

Hi @amyeroberts, PR here: https://github.com/huggingface/transformers/pull/27216 :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
fix:fix Deployment of web stream interface application, issue with me…
Author: bjleah
Pull request: yes

"When I deploy the Baichuan model using the streaming interface, I noticed that after each call, the GPU memory usage increases by more than ten gigabytes. Due to the inherent characteristics of the GPU, it's necessary to release the GPU memory during the token return process to ensure stable GPU memory usage."

Comments:
cc @gante 

Hi @bjleah 👋 

The current version of streaming in `generate` is a contraption that we want to properly replace in the near future. As such, we don't want to add more code in `generate` to handle a few special cases in streaming -- my suggestion would be to keep the changes you added in your local version of `transformers` (and point your project to your repository instead) :)

FYI, our next version of streaming will be a simple iterator returned from `generate`, from which the user can easily manipulate the tokens as they wish 🤗 

@gante Thank you， We'll look forward to the new streaming iterator in the next release. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
🌐 [i18n-ZH] Translate troubleshooting.md into Chinese
Author: yyLeaves
Pull request: yes

# What does this PR do?
Translate troubleshooting.md into Chinese
part of #20095  
## Who can review?
Documentation: @stevhliu 
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Report Tune metrics in final evaluation
Author: krfricke
Pull request: yes

# What does this PR do?

This PR makes Ray Tune's tuning objective function report all metrics, not just the objective, in the final evaluation step. It also gets rid of the (unnecessary) return value. With these changes the training objective is fully compatible with Ray Tune's recently introduced strict metric checking.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), 
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? 


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@sgugger 

Comments:
Just to be sure, is it also backward compatible with older versions of Ray Tune?

Yes! Return values were never required, and in fact disregarded until recently.

On October 1, 2020 2:38:00 PM GMT+01:00, Sylvain Gugger <notifications@github.com> wrote:
>Just to be sure, is it also backward compatible with older versions of
>Ray Tune?
>
>-- 
>You are receiving this because you authored the thread.
>Reply to this email directly or view it on GitHub:
>https://github.com/huggingface/transformers/pull/7507#issuecomment-702141017


Thanks for the fix then!

I'm confused about this PR. Why do we only evaluate once (metrics = local_trainer.evaluate()) and then report done=True directly? Some schedulers, like PopulationBasedTraining, need to evaluate many times during training to decide if the trial is good or bad.
Terminate TextIteratorStreamer Before Done
Author: fakerybakery
Pull request: no

Hi,
Is there any way to terminate a TextIteratorStreamer before the text has finished generating? Related to [this](https://github.com/gradio-app/gradio/issues/6724).
Thank you!

Comments:
Hey! I don't think this is possible through the TextIteratorStreamer but should be handled within gradio 🤗 

Ok, thanks for the suggestion! I'll close this issue then
Refactor and Enhance Readability of Zero-Shot Distillation in Transformers
Author: itsmenick212
Pull request: yes

# Description:

This pull request introduces a series of improvements to the HuggingFace Transformers library, specifically within the zero-shot distillation research project. The changes aim to enhance the code's readability, maintainability, and clarity through thoughtful refactoring and clear organization of dataset processing steps.



# What does this PR do?

With a focus on modularity and readability, this PR breaks down the monolithic main function into smaller, more manageable helper functions. The reorganization of dataset processing steps provides a clear path from data loading to tokenization, ensuring a more understandable and maintainable codebase.

Key improvements include:

Refactoring: The main function has been decomposed into several helper functions (initialize_student_model, create_compute_metrics_func, create_distillation_trainer, and train_and_evaluate_model), each with a distinct responsibility, improving modularity and readability.

Dataset Formatting: By grouping together dataset processing operations, the flow from data loading to its utilization in PyTorch is now more transparent and logical.

Helper Functions: New helper functions encapsulate specific tasks within the distillation process, making it easier to understand the workflow and modify individual components without affecting the overall system.

The changes made do not introduce additional dependencies and maintain the existing functionality while improving the structure and clarity of the code.

Comments:
Hi @itsmenick212, thanks for opening this PR and contributing to improving the repo! 

The research projects are [not maintained](https://github.com/huggingface/transformers/blob/8f1a43cd91cb22b65f1f840f6bca0e156e5e8495/examples/research_projects/README.md#L4) - as such we don't accept new PRs to update them. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
ImportError: cannot import name 'PegasusXForConditionalGeneration'
Author: wlmnzf
Pull request: no

### System Info

I follow the page(https://huggingface.co/docs/transformers/model_doc/pegasus_x) to try to use the PegasusXForConditionalGeneration.

But when I import the package ```from transformers import AutoTokenizer, PegasusXForConditionalGeneration```, it goes wrong.

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

>python3.6
>from transformers import AutoTokenizer, PegasusXForConditionalGeneration

### Expected behavior

I hope it an import the package correctly

Comments:
Hi @wlmnzf, thanks for raising this issue! 

It looks like the might be an issue with the installation of transformers in your environment. Please note that the lowest supported version of python is 3.8. I would suggest first try upgrading your python version and then reinstalling transformers. 

@amyeroberts Thanks for pointing about this, I try it again with python 3.10, and it works! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.