noamaanMulla-03/transformers-issues huggingface dataset:
enable optuna multi-objectives feature
Author: sywangyi
Pull request: yes

# What does this PR do?

Fixes # ((https://github.com/huggingface/transformers/issues/25657))


## Who can review?
@sgugger  @ArthurZucker 

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25969). All of your documentation changes will be reflected on that endpoint.

cc @muellerzr @pacman100 for input on the changes to trainer
Ability to pre-train BART model
Author: Hazoom
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

Ability to pre-train BART model, same like there is an ability to pre-train BERT and other models.

## Motivation

I'm using a pre-trained BART model for sequence-to-sequence problem and trained it using my own data, using the examples here: https://github.com/huggingface/transformers/tree/master/examples/seq2seq

I was wondering if there is a chance to add an ability to continue the pre-training of the already pre-trained `facebook/bart-base` and `facebook/bart-large` models, with my own unsupervised data, in order to improve the results.

@sshleifer Can you please help? thanks in advance!


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I was wondering if there was any follow up on this topic as I'd also be interested in continued pertaining on bart-base.
fix the deepspeed tests
Author: pacman100
Pull request: yes

# What does this PR do?
1. Post the https://github.com/huggingface/transformers/pull/25863, a test needs fixing. This PR does that

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Change itself look OK. 

Just for my own understanding, does this mean the trainer is changing variables - `a` in this instance - in its local scope, even if not explicitly passed to the trainer or set in any config? 

> Just for my own understanding, does this mean the trainer is changing variables - a in this instance - in its local scope, even if not explicitly passed to the trainer or set in any config?

The default value of the parameter is 0 and post the training the value should change, the latest commit makes it explicit.
Fine-tuning distilled GPT-2
Author: KerenzaDoxolodeo
Pull request: no

## ❓ Questions & Help

To my understanding, examples/run_lm_finetuning.py can be used to fine-tune the model to new data. How do I fine-tune a distilled GPT-2? To be precise, I assume that I can use the entire code, but I just need to import the right module. I tried importing DistilGPT2Config, DistilGPT2LMHeadModel, DistilGPT2Tokenizer, but it doesn't work out.

Comments:
_DistilGPT2Config_, _DistilGPT2LMHeadModel_ and _DistilGPT2Tokenizer_  **don't exist**. In order to fine-tuning the DistilGPT2 model for LM, you can use the following settings of tokenizer, config and model:
**Tokenizer**:
```
> from transformers import GPT2Tokenizer
> tokenizer = GPT2Tokenizer.from_pretrained('distilgpt2', )
```
N.B: as said in the source code [here](https://github.com/huggingface/transformers/blob/35401fe50fa3e460b2a4422630b017f106c79e03/transformers/tokenization_gpt2.py), this tokenizer requires a space to start the input string, therefore the `encoding` and `tokenize` methods should be called with the `add_prefix_space` flag set to `True`. Otherwise, this tokenizer's `encode`, `decode`, and `tokenize` methods will not conserve the spaces at the beginning of a string: `tokenizer.decode(tokenizer.encode(" Hello")) = "Hello"`

**Config**:
```
> from transformers import GPT2Config
> config = GPT2Config.from_pretrained('distilgpt2')
```

**Model**:
```
> from transformers import GPT2LMHeadModel
> model = GPT2LMHeadModel.from_pretrained('distilgpt2')
```
N.B: for completeness, in order to use DistilGPT2 model, you have to use the following code: `model = GPT2Model.from_pretrained('distilgpt2')`.

> ##  Questions & Help
> To my understanding, examples/run_lm_finetuning.py can be used to fine-tune the model to new data. How do I fine-tune a distilled GPT-2? To be precise, I assume that I can use the entire code, but I just need to import the right module. I tried importing DistilGPT2Config, DistilGPT2LMHeadModel, DistilGPT2Tokenizer, but it doesn't work out.

It works. Thank you.

I would like to know which code you are using for fine-tuning
Fix `MarianTokenizer` to remove metaspace character in `decode`
Author: tanaymeh
Pull request: yes

# What does this PR do?
This PR fixes the `MarianTokenizer` so that it removes the metaspace character during decode (`▁`).

Fixes #26018

## Who can review?
@ArthurZucker, @xenova 

Comments:
> make sure the fast tokenizers also works!

Unfortunately, there's no `MarianTokenizerFast` 😅  

Thanks for the review, @ArthurZucker!
I added the following test (it checks with starting special characters in a string like an underscore).

```python
def test_tokenizer_decode(self):
    tokenizer = MarianTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-es")
    source_text = "_This is 1 text string that starts with an _ and ends with one too _"
    ids = tokenizer(source_text)["input_ids"]
    output_text = tokenizer.decode(ids, skip_special_tokens=True)
    self.assertEqual(source_text, output_text)
```

Does it look good to you?

> > make sure the fast tokenizers also works!
> 
> Unfortunately, there's no `MarianTokenizerFast` 😅

@xenova @ArthurZucker I would love to add the `MarianTokenizerFast` if that is considered a worthwhile addition to 🤗 transformers!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26091). All of your documentation changes will be reflected on that endpoint.

Thanks for the reviews and merging!
I was wondering @ArthurZucker if `MarianTokenizerFast` would be a worthwhile addition to HF transformers and if I can contribute to adding it.

It does not seem to have been requested a lot 😄  but feel free to add it if you want some good experience with `tokenizers`

I think this would be a good addition, simply because of the number of monthly downloads the >1400 models get 😇. The top 5 models alone total ~2.75M downloads in the past month.

https://huggingface.co/models?sort=downloads&search=Helsinki-NLP%2Fopus-mt
Fix AutoTokenizer docstring typo
Author: amyeroberts
Pull request: yes

# What does this PR do?

Fixes #26108


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._
The "padding" parameter of tokenizer doesn't seem to work
Author: a815063199
Pull request: no

### System Info

- `transformers` version: 4.33.0
- Platform: Linux-4.15.0-206-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker
The padding parameter of the tokenizer seems to have no effect, and the tensor shape returned by the tokenizer using padding is not being padded. Use pad_to_max_length parameter can be correctly padding


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from transformers import BertModel, BertTokenizer, BertTokenizerFast

model_name = 'moka-ai/m3e-base'
tokenizer = BertTokenizerFast.from_pretrained(model_name)

sentences = "我今天中午吃的拉面，你吃的啥"

inputs = tokenizer.encode_plus(
    sentences,                           # 输入文本
    add_special_tokens=True,        # 添加特殊标记
    max_length=512,                 # 最大长度
    #pad_to_max_length=True,          # 使用padding填充
    padding=True,          # 使用padding填充
    truncation=True,
    return_attention_mask=True,      # 返回attention mask
    return_tensors='pt'              # 返回pt类型的张量
)

for k, v in inputs.items():
    print(v.shape) # except shape is (1, 512), but get (1, 16) when using "padding=True"
```

### Expected behavior

the padding parameter of tokenizer should work same as the pad_to_max_length

Comments:
Hi @a815063199, thanks for raising this issue! 

You need to pass `padding="max_length"` to pad to the max length specified. With `padding=True` the tokenizer pads to the max sequence length in the batch. There's more information in the docs here: https://huggingface.co/docs/transformers/pad_truncation

ok～ i misunderstood the meaning of `padding=true`. thanks`
[`core`] fix 4bit `num_parameters`
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/25978

For 4bit models, the parameters are stored under `bnb.Param4bit` class, that uses `torch.uint8` tensors to store the 4bit weights. However, since the uint8 precision is in 8bit, half of its storage is used. To give a more concrete example, if we want to store a 100x100 matrix in 4bit using `Param4bit`, it will be stored under a 100x50 Parameters in uint8. This is what I know per my understanding, therefore when computing the number of parameters it is crucial to multiply the number of elements `numel()` by two, to retrieve the correct number of elements.

With this PR, we compute the correct number of parameters for 4bit models, and a nice test has been added.

cc @amyeroberts and @SunMarc just for your information in case you were not aware of this 


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Update spectrogram and waveform model mapping for TTS/A pipeline
Author: Vaibhavs10
Pull request: yes

# What does this PR do?

Updates the mapping names for the models used for TTS & TTA pipelines.

Fixes # (issue)
`MODEL_FOR_TEXT_TO_SPECTROGRAM_NAMES` -> `MODEL_FOR_TEXT_TO_SPECTROGRAM_MAPPING_NAMES`
`MODEL_FOR_TEXT_TO_SPECTROGRAM_NAMES` -> `MODEL_FOR_TEXT_TO_WAVEFORM_MAPPING_NAMES`

## Who can review?

@ylacombe @sanchit-gandhi 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sanchit-gandhi I just tested, and I essentially get no output!
Do you think that's good?

Requesting final review from Arthur! (who I believe is now on watch 👀)
Create tests for tiny Bark
Author: ylacombe
Pull request: yes

Following @ydshieh's #25290 which laid the groundwork for tiny Bark tests, this PR adds BarkModel tests.

Because BarkModel is a non-regular model, with no forward method and a non-regular `generate`, I've created some hand-made tests, which I can enrich further if you like. @ydshieh , @amyeroberts  WDYT of the tests already implemented?

Many thanks!


Comments:
_The documentation is not available anymore as the PR was closed or merged._

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

We could close this (at least for now) if @ylacombe is fine. The original purpose (from my side) is no longer valid.

Totally agree!
[`RWKV`] Final fix RWMV 4bit
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/23848

Double quantization was not working properly for RWKV models as stated in the PR above - leading to an error. This PR proposed a global fix for RWKV models so that they can be ran in 4bit bitsandbytes without any problem.

The followed approach here is the following: 

- For each target layer, de-quantize the 4bit weights using `bnb.nn.functional.dequantize_4bit`
- Perform the weights scaling
- Requantize the weights again.

That way it is possible to make sure to cover both the double quantization and classic 4bit quantization and match the results together

```python
import torch
from transformers import RwkvForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_id = "RWKV/rwkv-4-169m-pile"

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True
)

model = RwkvForCausalLM.from_pretrained(model_id, quantization_config=quantization_config)
tok = AutoTokenizer.from_pretrained(model_id)

text = "Hello my name is"
input_ids = tok.encode(text, return_tensors="pt").to(0)

out = model.generate(input_ids, max_new_tokens=30)
print(tok.decode(out[0], skip_special_tokens=True))

model_non_dequant = RwkvForCausalLM.from_pretrained(model_id, load_in_4bit=True)

text = "Hello my name is"
input_ids = tok.encode(text, return_tensors="pt").to(0)

out = model.generate(input_ids, max_new_tokens=30)
print(tok.decode(out[0], skip_special_tokens=True))
```

cc @amyeroberts and @SunMarc for your information!

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for the review, I added a test that should be applicable to other checkpoints as well, as they use the same arch
docs: add LLaMA-Efficient-Tuning to awesome-transformers
Author: ji-huazhong
Pull request: yes

# What does this PR do?

Kindly ask to add [LLaMA-Efficient-Tuning](https://github.com/hiyouga/LLaMA-Efficient-Tuning) to the list of awesome projects that has transformers support

## Who can review?
cc @sgugger
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25441). All of your documentation changes will be reflected on that endpoint.
Improve training args
Author: ji-huazhong
Pull request: yes

# What does this PR do?
This PR has two main modifications:
~~1. the `per_gpu_train/eval_batch_size` training args have been removed as they were deprecated in #4618 three years ago. In my opinion, it is appropriate to remove them.~~
2. the information regarding certain training args has been updated for better promptness. This is because Transformers now provides support for multiple accelerators, including cuda, tpu, mps, and npu.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Thanks for iterating!
> 
> Can you just run `make style` on your branch to fix the quality issue?

@sgugger ready to merge:)
replace no_cuda with use_cpu in test_pytorch_examples
Author: ji-huazhong
Pull request: yes

### What does this PR do?
This PR replace `no_cuda` with `use_cpu` in `test_pytorch_examples.py`, as `no_cuda` training argument is deprecated([see](https://github.com/huggingface/transformers/pull/24863)) . By the way delete a piece of code that will never be used.


Comments:
_The documentation is not available anymore as the PR was closed or merged._
fix a typo in docsting
Author: ji-huazhong
Pull request: yes

# What does this PR do?

as per title


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerz and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25759). All of your documentation changes will be reflected on that endpoint.
Remove deprecated codes
Author: ji-huazhong
Pull request: yes

# What does this PR do?


This PR removes some deprecated code:
- remove `xpu_backend` training argument, for it is deprecated and will be remove in version 4.31 of Transformers.
- remove some codes that will never be executed.



## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@sgugger

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
It seems the failing test case is not related to these commits. Please re-trigger this failed workflow, Thanks.

_The documentation is not available anymore as the PR was closed or merged._
docs: feat: add llama2 notebook resources from OSSCA community
Author: junejae
Pull request: yes

# What does this PR do?
Adds a notebook resource of LLaMA2 on how to fine-tune LLaMA2 as a Korean text classifier.
The notebook is created by our OSSCA community.
Part of https://github.com/huggingface/transformers/issues/20055

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?
May you please review this PR?
@bolizabeth, @nuatmochoi, @heuristicwave, @mjk0618, @keonju2, @harheem, @HongB1, @junejae, @54data, @seank021, @augustinLib, @sronger, @TaeYupNoh, @kj021, @eenzeenee


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26076). All of your documentation changes will be reflected on that endpoint.

> Nice, thanks for creating a notebook! 🤗
> 
> I wasn't able to get the code in it to run successfully though. I get the following error because I think in your implementation of the `TrainingArgument` class you have `batch_size` instead of `per_device_train_batch_size`?
> 
> ```python
> TypeError: TrainingArguments.__init__() got an unexpected keyword argument 'per_device_train_batch_size'
> ```

@stevhliu 
It's fixed! Could you try it again?

I messed up my code while polishing some variable names.
I rolled back the change and tested. It works fine now!

Cool, it works now; feel free to mark this as ready, and then we can merge!
Generate: ignore warning when `generation_config.max_length` is set to `None`
Author: gante
Pull request: yes

# What does this PR do?

As raised and suggested by our fellow @BramVanroy, ignores the warning complaining about both `max_new_tokens` and `max_length` being set when `generation_config.max_length is None`

This tackles needless warnings when users set the generation config like (e.g.):
```
gen_config = GenerationConfig.from_pretrained(
    model_name,
    max_new_tokens=200,
    max_length=None,  # because the model has some `max_length` defined and we don't want it
    num_beams=1,
)
```

Comments: