noamaanMulla-03/transformers-issues huggingface dataset:
[docstring] Fix docstring for CanineConfig
Author: Sparty
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes https://github.com/huggingface/transformers/issues/26638


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Nice it works for you now @Sparty . One minor change and we are ready to go 🚀 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26771). All of your documentation changes will be reflected on that endpoint.
Add Japanese translation
Author: shinshin86
Pull request: yes

# What does this PR do?
Hello!

I found a description in `README_ja.md` that is not translated into Japanese.
Therefore, I am only translated the relevant sections.

Thank you!

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Documentation: @stevhliu and @MKhalusova


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26799). All of your documentation changes will be reflected on that endpoint.
Make `ModelOutput` serializable
Author: cbensimon
Pull request: yes

Currently, `@dataclass` `ModelOutput` instances can't be pickled, which can be inconvenient in some situations
This PR fixes this by adding a custom `__reduce__` method to the `ModelOutput` class

Original PR from diffusers : https://github.com/huggingface/diffusers/pull/5234

**EDIT**: Actual use case for me is passing a ModelOutput instance in a multiprocessing queue
(this is needed if a model `__call__` is wrapped inside the ZeroGPU decorator : `model = spaces.GPU(model)`)

Comments:
_The documentation is not available anymore as the PR was closed or merged._


> passing a ModelOutput instance in a multiprocessing queue

I have experience regarding things become slow (or even blocked) when passing (large) tensors between processes. 

But maybe things change along time and this is no longer a common issue.

> > passing a ModelOutput instance in a multiprocessing queue
> 
> I have experience regarding things become slow (or even blocked) when passing (large) tensors between processes.
> 
> But maybe things change along time and this is no longer a common issue.

Sure, the PR is more here by correctness / symmetry because the same thing is done on diffusers side (but I agree it makes more sense in diffusers since it's a pipeline output while it's a model output in transformers)
Make fsdp ram efficient loading optional
Author: pacman100
Pull request: yes

# What does this PR do?
1. Make fsdp ram efficient loading optional. Certain models are having issues when handling meta devices during pre-trained model loading. Fixes https://github.com/huggingface/accelerate/issues/1948 and https://github.com/huggingface/accelerate/issues/2031 by making the ram efficient loading optional.
2. This PR should be merged after PR https://github.com/huggingface/accelerate/pull/2037 is merged.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

That PR is merged @ArthurZucker, Thank you!
fix resume_from_checkpoint bug
Author: Jintao-Huang
Pull request: yes

# What does this PR do?

Fixes bug: 
When `resume_from_checkpoint` is set to True, the `state` is imported, which includes `self.state.best_model_checkpoint`. If the best model is still the one corresponding to the model in the resume_from_checkpoint directory when saving the model during training, an exception will be raised at `best_model_index = checkpoints_sorted.index(str(Path(self.state.best_model_checkpoint)))`: 
ValueError: '/path/to/checkpoint-100' is not in list.

Fixes issue(https://github.com/huggingface/transformers/issues/22172)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

- trainer: @muellerzr and @pacman100

Comments:
Hello, it seems that citest has encountered a timeout error. It doesn't seem to be an error on my end. Can you help me take a look, please?

Hi, any chance you can rebuild/rebase from main? I think that will fix your failing tests. Thanks!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26739). All of your documentation changes will be reflected on that endpoint.
`prefix_allowed_tokens_fn` do not constrain when all allowed tokens have scores of `-inf`
Author: ksh108405
Pull request: no
Labels: name: WIP

### System Info

transformer 4.25.1
python 3.8.16

### Who can help?

@gante

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

When using `generate()` with `prefix_allowed_tokens_fn`, (more precisely, when using `PrefixConstrainedLogitsProcessor`),
when all tokens returned by `prefix_allowed_tokens_fn` have scores of `-inf`, the model does not comply with the constraints and picks the token which is not on the allowed token list.

### Expected behavior

Even if all allowed tokens have score of `-inf`, the model should pick tokens from allowed token list by `prefix_allowed_tokens_fn`.
I think it can be solved by using some clamp function or adding epsilon value to this code.
https://github.com/huggingface/transformers/blob/04ab5605fbb4ef207b10bf2772d88c53fc242e83/src/transformers/generation/logits_process.py#L692-L698

This is my own code to solve it.
However, it might cause other bugs.

```python
    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor) -> torch.FloatTensor:
        masked_score = torch.full_like(scores, -math.inf)
        for batch_id, beam_sent in enumerate(input_ids.view(-1, self._num_beams, input_ids.shape[-1])):
            for beam_id, sent in enumerate(beam_sent):
                allowed_idx = batch_id * self._num_beams + beam_id, self._prefix_allowed_tokens_fn(batch_id, sent)
                filtered_scores = torch.clamp(scores[allowed_idx], min=-10 ** 6)
                masked_score[allowed_idx] = filtered_scores
        return masked_score
```
Edit: The model works well on `torch.clamp()` with `min=-10 ** 6`, not `min=-10 ** 8`, when all allowed token's score is -inf. Too low score token in the sequence may have affected the decoding step. I updated the above code.

Comments:
Hey @ksh108405 👋 

Constrained generation has several issues at the moment, and I'm out of bandwidth. I'm adding this to the list of things keep an eye on when revisiting constrained generation :)

Hello @gante , I'm happy to work on this issue. Do you approve?

(I have some experience with constrained decoding previously)



Hey @Saibo-creator 👋 

Of course, we always welcome contributions to fix issues 💛 Thank you for offering help!
Add LLM doc
Author: patrickvonplaten
Pull request: yes

# Optimize LLM tutorial

This PR adds the doc version of https://huggingface.co/blog/optimize-llm .
Given the many code snippets I think that the tutorial is a nice addition to the Transfomers docs.

Comments:
Doc will also be released with blog post: https://github.com/huggingface/blog/pull/1473

_The documentation is not available anymore as the PR was closed or merged._

Thanks for the review @MKhalusova! 

Answering the bullet points directly in line:

> Given the level of detail, and the narrative style, I think the doc would fit better under the Conceptual Guides section rather than in Tutorials.

Happy to move it to the conceptual guides if it fits better - @gante wdyt?

> I would suggest changing the title of the doc as it is not very descriptive. Perhaps "LLM inference optimizations"/"Efficient LLM deployment"?

Yes, agree. Should be better now.

> It would also be great to shorten/simplify the guide where possible.

I'd need more precise comments here. We could maybe also do this in a second pass. As a general guide on how to optimize LLMs for memory and speed, I'm not sure how I would simplify or shorten it.

> There are some really cool gems in the doc that would be great to highlight somehow (bold font or using a <Tip>), e.g. "Therefore, inference time is often not reduced when using quantized weights, but rather increases".

Guess this is also a style issue. Maybe this can be done in a second pass to align it more with other docs.

> I have left some nits regarding style, issues with formula rendering, etc.

Addressed them

@gante could you also take a look here?
Llama tokenizer: remove space in template comment
Author: pcuenca
Pull request: yes

I think the space between the eos and bos tokens is not present in the actual template output. I'm using this documentation as a reference for everyone asking about prompting, so would like to clarify whether there's a space or not :)

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ Rocketknight1, @ArthurZucker 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Our template gives identical results as llama 2's original tokenization, I just double-checked by comparing the tokens using a three-turn conversation. The code has always been correct, there's no space between the `eos`/`bos` tokens.

Not sure what you mean by _linking to meta's original code_, should we include a link in the docs? This would be the place where the user/assistant turns are parsed: 
https://github.com/facebookresearch/llama/blob/556949fdfb72da27c2f4a40b7f0e4cf0b8153a28/llama/generation.py#L342


Yep if you feel like it! This way they can compare with their own eyes the source of truth! (even if it's code)

Thanks to you both for your quick reviews! Feel free to merge when you see fit, I don't have write permissions in this repo :)

Done!
Enable testing against mi250
Author: mfuntowicz
Pull request: yes

Currently we are testing only against mi210 AMD devices, this PR enables testing on mi250 as well 🤗

Comments:
_The documentation is not available anymore as the PR was closed or merged._

[**Update**]

I might be able to do minimal changes to make the slack report work with the CI having 2 GPU favors like this PR. I will have to try though.

----------------------------------------------


As testing itself, this change is OK.

However, this will **break the slack reporting**, as both `mi210` and `mi250` jobs will produce the same artifacts. The file in `utils/notification_service.py` (slack reproting) is designed to work with a workflow run that has run the CI once. (In this PR, it's kinda twice).

And even if we have different artifact names (together some changes in `utils/notification_service.py` so we can handle them), we still need to **reproduce the tables for both**  `mi210` and `mi250` together with all the **failing tests as slack replies**. This is going to difficult to navigate.

- It also makes the workflow run (a bit) harder to navigate as now it contains twice more jobs
- Make change to `utils/notification_service.py` will make that file being harder to understand/maintain.

A quick, simple way (despite not ideal) is to create another workflow for  `mi250`. I know it's duplicated (and we don't like), but everything has advantage and disadvantage.

@mfuntowicz We will have 3 GPU favors or even more?
@LysandreJik : WDYK? Duplicated workflow file so we don't need to rework on slack reporting script, or another direction?



Yes 3 flavors, nothing more expected so far 🙂

From my POV it seems a bit overkill to duplicate the whole workflow file "just" to handle multiple instance type (i'm pretty sure in the future you'll also want to test T4/H100/etc.). Also the increased maintenance cost and accounting for the probability to forgot to update one of the file 😁.

My 2cents 🙂.

Your points are also valid :-) I have to make the changes to the slack reporting script. Could this PR wait a bit?



~~@mfuntowicz In order not to block this PR, let's go for duplicating. Eventually, I can rework when we decide a change is necessary at some point.~~

I might have some way to do it better. Let me check

For @mfuntowicz to check #26634. If it is fine, we can incoperate the changes into this PR.

 @mfuntowicz any comment for #26634 ?

Just approved it, thanks a lot @ydshieh 🙏🏻.

LMK if there is anything i need to take a look at to merge these PRs :) 

Close in favor of #26634. Will add mfuntowicz as the contributor.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26555). All of your documentation changes will be reflected on that endpoint.
How-to-fine-tune-bert-for-question-answering?
Author: anirbansaha96
Pull request: no
Labels: name: Help wanted

I wish to train two domain-specific models:

Domain 1: Constitution and related Legal Documents
Domain 2: Technical and related documents.
For Domain 1, I've access to a text-corpus with texts from the constitution and no question-context-answer tuples. For Domain 2, I've access to Question-Answer pairs.

Is it possible to fine-tune a light-weight BERT model for Question-Answering using just the data mentioned above?

If yes, what are the resources to achieve this task?

Some examples, from the huggingface/models library would be mrm8488/bert-tiny-5-finetuned-squadv2, sshleifer/tiny-distilbert-base-cased-distilled-squad, /twmkn9/albert-base-v2-squad2.

[You can find the question here.](https://datascience.stackexchange.com/questions/77213/how-to-fine-tune-bert-for-question-answering)

Comments:
Hi, how did you solve this project? I want to do the same in another domain and I want to know what is the best approach.
Conversation pipeline fixes
Author: Rocketknight1
Pull request: yes

This PR makes a couple of fixes to `ConversationalPipeline` to make it a lot easier to use:

- Inputs can now just be conversations in standard list-of-dicts format. I think the `Conversation` class is quite hard for users to discover, and this is a lot more intuitive.
- We no longer read `max_length` because very few models set this parameter, and so it's almost always the default `PretrainedConfig` value of 20, which is very low. Before this change, most calls to `ConversationalPipeline` produced no output or unnecessarily truncated the input because this limit was hit. We change the pipeline to use `max_new_tokens` instead, which is more modern.

cc @arthurzucker for pipeline review and @gante if he has any comments about setting the generation parameters properly!

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Also cc @lewtun - now you should actually be able to just use this pipeline in the docstrings instead of needing to do it manually in the docstrings with `text-generation` and `apply_chat_template`!
Adding new special tokens does not seem to work properly in 4.34
Author: dakinggg
Pull request: no

### System Info

- `transformers` version: 4.34.0
- Platform: macOS-13.5-arm64-arm-64bit
- Python version: 3.10.12
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: 0.20.3
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

In [10]: import transformers

In [11]: llamat = transformers.AutoTokenizer.from_pretrained('meta-llama/Llama-2-7b-hf')

In [12]: llamat('hello')
Out[12]: {'input_ids': [1, 22172], 'attention_mask': [1, 1]}

In [13]: llamat.add_special_tokens({'bos_token': '[NEWSPECIAL]'})
Out[13]: 1

In [14]: llamat('hello')
Out[14]: {'input_ids': [1, 22172], 'attention_mask': [1, 1]}

### Expected behavior

Expected that the new bos token is used instead of the old one.

Comments:
@ArthurZucker 

Hey, this is expected for a fast tokenizer because you are not changing the `processor` . You need to run `llamat.update_post_processor()` before to make sure this is taken into account. 
Initializing the tokenizer would be the best way to have the expected results for fast tokenizers


Thanks, some clarifications:
(1) Is it expected that this behavior changed in 4.34? Its a bit scary of a behavior to silently change and could lead to very tricky bugs in peoples' workflows.
(2) Is the recommendation that I always call `update_post_processor` after adding new special tokens?

1. It's not necessarily from 4.34. I tested with 4.33 and this does not work either. 
2. Fast tokenizers as supposed to be stateless, and we usually don't support post init updates. My recommendation is rather to do everything at init time rather than post init for fast tokenizers 😉 

Huh, I wonder why my unit test was passing before 😅 but anyway, makes sense, thank you!