noamaanMulla-03/transformers-issues huggingface dataset:
QAT of segformer
Author: TanyaChutani
Pull request: yes

# What does this PR do?
QAT of segformer

## Who can review?
- PyTorch: @sgugger
- vision models: @amyeroberts

Comments:
Hi @TanyaChutani. thanks for opening this PR! 

We won't accept this change as it is. We can't just replace layers like this in the library, as it will break many things for our current users. In particular, it will automatically quantize the modules which we don't want to do. 

Some quantization functionality is already available within transformers using bitsandbytes. Check out relevant resources here : 
* https://huggingface.co/docs/transformers/main_classes/quantization
* https://huggingface.co/blog/4bit-transformers-bitsandbytes

Hi @amyeroberts, thanks for a quick reply. 

Is it possible to add an argument `_quantize` - with the help of it we can utilize both quantize and non-quantized layers? 

For instance, 
```
if _quantize:
     quant_nn.QuantConv2d()
nn.Conv2d()
```

Please do let me know your thoughts on it.

As I mentioned above, quantization is already possible within transformers and accelerate: 
* https://huggingface.co/docs/transformers/main/en/main_classes/quantization
* https://huggingface.co/docs/accelerate/v0.21.0/en/usage_guides/quantization

Have you tried using these approaches to quantize your model? Is there something being added in this PR which isn't currently possible with these approaches? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Put Falcon back
Author: LysandreJik
Pull request: yes

Goes hand in hand with https://github.com/huggingface/transformers/pull/25954

As seen offline with @ArthurZucker but would still like additional eyes on this as it's touching critical code (that I'd honestly rather not be touching, but I don't see another way around it).

This code will be removed as soon as we can revert the change on the Falcon repositories as `from_pretrained` will then automatically download the appropriate revision.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[MusicGen] Add sampling rate to config
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Adds sampling rate attribute to the parent model config. We get the sampling rate from the EnCodec sub-model config. Adding this attribute brings consistency with the other TTA models in the lib, where the output sampling rate is accessible directly through the top-level config (https://github.com/huggingface/audio-transformers-course/pull/140#discussion_r1324226366)


Comments:
_The documentation is not available anymore as the PR was closed or merged._
[Whisper] Check length of prompt + max new tokens
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Fixes #25422: adds a check for the combined length of prompt + max new tokens. If this total exceeds the model's max length (`max_target_positions`), we throw an error.

cc @connor-henderson @Helene-Maxcici 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Trainer explodes with multiple validation sets used
Author: radekosmulski
Pull request: no

### System Info

- `transformers` version: 4.32.0.dev0
- Platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.35
- Python version: 3.11.4
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- Accelerate version: 0.22.0.dev0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.0.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no, running in notebook
- Using distributed or parallel set-up in script?: no, running in notebook

### Who can help?

@sgugger 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Using multiple datasets, the trainer looks for a key that doesn't exist and throws an error upon evaluation at the end of the epoch.

![image](https://github.com/huggingface/transformers/assets/2444926/bbc98fb0-b58d-42e0-8992-fef2139c5ad6)


### Expected behavior

Metrics are returned just fine. There is no error.

Comments:
Yes, multiple evaluation datasets are not supported in a notebook env, that is a known issue.

Thank you, Sylvain! 🙂 Didn't know that! Appreciate your reply, will close this now.

Ah ah you don't need to close it ;-) It's not been high-priority for us but we should fix it at some point.

Ok, sorry Sylvain 😊 Let me reopen the issue then! I thought it was a known issue as in "we know it doesn't work but it is meant to be broken in notebooks" and I just didn't realize that, but if it is a genuine issue then let me leave it open! 🙂

Will ping @pacman100 here to put this on radar for the trainer! 

More for @muellerzr, but he should already have it somewhere in his TODO for when he has time ;-)

#25796 was opened to fix it
Error with resize_token_embeddings and LLama model
Author: lccnl
Pull request: no

### System Info

- `transformers` version: 4.32.0
- Platform: Linux-5.15.0-1038-gcp-x86_64-with-glibc2.31
- Python version: 3.10.7
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.2
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

In the last version of the method `resize_token_embeddings`, the `pad_token_to_multiple=N` arg allows to increase the dimension to the first  int divisibile by N larger than `new_num_tokens`. 

However, in the method, the `config.vocab_size` is still updated to the initial [new_num_tokens](https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py#L1438).

For the LLama model, this leads to an error when the logits are flattened [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L837). 



### Expected behavior

We should update the vocab_size to the new_num_tokens that accounts for the arg pad_token_to_multiple

Comments:
cc @SunMarc Maybe ?

Non that's actually me, I'll open a PR to fix this
Fix pad to multiple of
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fixes #25729, `pad_to_multiple_of` did not overwrite the `model.config.vocab_size` properly.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Will fix the test, if model is not tied, we should not update the embedding, but that is also `True` for all models !  

Thank you @ArthurZucker for this! I am currently being bitten by #19418, so I look forward to this being merged!

I'll try to merge this asap 😉 sorry for the trouble!
[docs] IDEFICS guide and task guides restructure
Author: MKhalusova
Pull request: yes

There are two now common approaches to solving tasks: fine-tuning specialized models, and prompting large models. While the fine-tuning approach is prominently featured in the docs, the latter one is not. At the same time, generation docs fall on the sidelines. 

This PR includes two parts: 
1. A guide to using IDEFICS for various image-text tasks such as image captioning, vqa, etc. This helps not only highlight the non-fine-tuning approach in the docs, but also showcase the new open source model.
2. TOC restructure. In this PR, I suggest a restructured table of contents where the task guides are now called "Guides for fine-tuning specialized models", and there's a new section on the same level called "Prompting and Generation Guides for Large Models". This new section currently contains the IDEFICS guide, and the "Text generation strategies" doc. In the future, we can add prompting guides, and updated text generation docs here. 
The new structure reflects two different approaches to tasks. 
Currently the zero-shot guides for specialized models are still under fine-tuning task guides, and we can discuss whether they should be moved under the new section or not. 

Comments:
PR with images for the doc: https://huggingface.co/datasets/huggingface/documentation-images/discussions/175

_The documentation is not available anymore as the PR was closed or merged._

@LysandreJik Please suggest who else should be added as a reviewer. Thanks!

Think this is a bit in conflict with this PR: https://github.com/huggingface/transformers/pull/25371

I'm not a huge fan of changing the more general "Task Guide" title to the more specific "Fine-tuning Guide" because I think many of the guides contain very valuable information that's just needed for inference.

E.g. let's say right now I want to know how to use a fine-tuned speech model from the Hub for speech transcription, I wouldn't necessarily look under "Fine-tuning Guides"
To be it would make more sense to either:
- Leave "Task Guide" as a title and integrate LLM prompting and Idefics under it
- Or completely split inference and training into two different areas of the docs.

Not sure what @gante thinks here

re @patrickvonplaten 's concerns, I've reverted the TOC structure reorg, and placed the IDEFICS guide under Task Guides. 
I've also addressed @LysandreJik 's feedback. 

Amazing! thank you! looking at it today!
[PEFT] Allow PEFT model dict to be loaded
Author: patrickvonplaten
Pull request: yes

In order to allow `peft` to be leveraged in `diffusers` without breaking changes we need to allow loading adapters directly from a loaded `state_dict`. The reason is that in `diffusers` we currently store LoRA checkpoints in a format that is different to the PEFT format so we cannot just pass the model_id. This PR allows the user to manually pass a loaded PEFT model checkpoint as well as a PEFT configuration, thus circumventing the need to pass a model id.

In pseudo code, the integration of `transformers` + PEFT in `diffusers` should then look as follows the ["load_lora"](https://github.com/huggingface/diffusers/blob/24c5e7708bb75076dd8e79ccaea195640555f945/src/diffusers/loaders.py#L1213) function of `diffusers`.


```py
def load_lora_into_text_encoder(cls, state_dict, network_alphas, text_encoder, prefix=None, lora_scale=1.0):
    peft_state_dict, peft_config = convert_to_peft_format(state_dict, ...)  # <- this function will take care of all the remapping necessary for the different formats
    text_encoder.load_adapter(peft_state_dict, peft_config=peft_config)
```

**Note**, there might be more changes we have to do to PEFT, `transformers'` PEFT integration to be sure that everything works as expected. E.g. I'm not yet sure how to pass `network_alphas` etc... to PEFT to make sure we get 1-to-1 the same result.

cc @younesbelkada @sayakpaul @BenjaminBossan @pacman100 
    

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Error with load_in_4bit argument when loading model
Author: matsuobasho
Pull request: no

### System Info

OS: Windows 10 64-bit
Python version: 3.9.6
Transformers: 4.33.1  (imported with `pipenv install transformers[torch]`)
Accelerate: 0.22.0
BitsandBytes: 0.41.1

Environment manager: pipenv

### Who can help?

@SunMarc @younesbelkada 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I get an error saying that I need Accelerate when using the `load_in_4bit` argument in the `from_pretrained` method.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

checkpoint = "Salesforce/codegen-350M-mono" 
device = "cpu"

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(checkpoint, trust_remote_code=True, load_in_4bit = True).to(device)
```

Error:
```ImportError:` Using `load_in_8bit=True` requires Accelerate: `pip install accelerate` and the latest version of bitsandbytes `pip install -i https://test.pypi.org/simple/ bitsandbytes` or pip install bitsandbytes` ```

Same issue expressed by many people in [this](https://huggingface.co/anon8231489123/vicuna-13b-GPTQ-4bit-128g/discussions/11) post.  

### Expected behavior

Model loads without errors.  

Comments:
Hi @matsuobasho , I can't reproduce the error after switching to your setup. Please check which one of these conditions fails: `is_accelerate_available()` and `is_bitsandbytes_available()`. You can import them with 
```py
from transformers.utils import is_accelerate_available, is_bitsandbytes_available
```

@SunMarc, thanks for the response.

`is_bitsandbytes_available()` returns False

Then check inside the code where it returns false. It is either torch or bitsandbytes the problem. After you find it, reinstall the library. 

I'm able to import torch.
When I import bitsandbytes I see this warning:
`UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
'NoneType' object has no attribute 'cadam32bit_grad_fp32'`

So that seems to be the problem but not sure how to resolve.  Also, do I need CUDA in order for bitsandbytes to work?  I'm just running this locally on a CPU.

Hi @matsuobasho 

> So that seems to be the problem but not sure how to resolve. Also, do I need CUDA in order for bitsandbytes to work? I'm just running this locally on a CPU.

Note that bitsandbytes features we have integrated only works on GPU, so your behaviour is expected unfortunately. To use these feature, you can use for instance free-tier GPU instances such as google colab or kaggle notebooks

Ok, thank you.
BitsAndBytes
Author: Shikamaru5
Pull request: no

I'm trying to run my model with:

    bnb_config = BitsAndBytesConfig(
                load_in_4bit=True,
                bnb_4bit_quant_type="nf4",
                bnb_4bit_compute_dtype=torch.float16,
                )
Trouble is when I run it I get an error about some modules are dispatched on the CPU or the disk. If you want to dispatch the model on the CPU or the disk while keeping these modules in 32-bit, you need to set `load_in_8bit_fp32_cpu_offload=True` and pass a custom `device_map` to `from_pretrained`
Looking into it on hugging_face and goodle, and some of the code in modeling_utils.py. I didn't see anything that suggested you could do something like 'load_in_4bit_fp32_cpu_offload=True' or something along those lines. I'm guessing it isn't a feature yet, or maybe coming? If anyone has any ideas about that I'd be really grateful. I'm going to test the 8bit thing and see if that's enough but I'm not totally certain that 8bits will be low enough to load the model unfortunately.

Comments:
Hey 🤗  thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help!
You can also ping @younesbelkada there. 
Thanks!

Ok sounds good thanks, I just added the question on the forum.

+1 I amended the forum discussion but to me looks more like a bug.

cc @younesbelkada maybe just update the warning to include `4bits` 

I'm not too sure but I added this to bnb_config: load_in_8bit_fp32_cpu_offload=True and then I created a custom device_map with every single layer of the model to the gpu which ended up being around 456 layers, but it seems to have loaded now, yay me!

well if we specifically don't want any kind of offload (hence leave this flag to False) then I think there is no reason why this warning should pop up, but maybe I am missing something.

Hey there! 
I think passing `load_in_8bit_fp32_cpu_offload` also works for 4bit models, if that's the case I agree it is confusing, we should add a new arg `load_in_4bit_fp32_cpu_offload` that would behave exactly as the 8-bit arg. I can work on that and propose a fix. I will keep you posted
Issue: sacremoses library missing
Author: Konjarla-Vindya
Pull request: no

### System Info

Hi @ArthurZucker, we are using AutoTokenizer for importing HuggingFace models into AzureML. This gave an error when importing 'xlm-mlm-en-2048' model. The error says ModuleNotFoundError: No module named 'sacremoses'

Error snip

![image](https://github.com/huggingface/transformers/assets/137049777/591fa265-3ac0-4a6f-95cc-ffaba08da8fe)


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

%pip install transformers[torch]
%pip install torchvision
%pip install mlflow
%pip install python-box

from transformers import AutoModel, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("xlm-mlm-en-2048")
model = AutoModel.from_pretrained("xlm-mlm-en-2048")

### Expected behavior

xlm-mlm-en-2048 should work with AutoModel

Comments:
Hey! could you share the `transformers` version you are using? 
The error should be the following: 
```python 
        try:
            import sacremoses
        except ImportError:
            raise ImportError(
                "You need to install sacremoses to use XLMTokenizer. "
                "See https://pypi.org/project/sacremoses/ for installation."
            )
```
see [here](https://github.com/ArthurZucker/transformers/blob/f4b3c85fa8a351e336d7a02f96cef781550821ff/src/transformers/models/xlm/tokenization_xlm.py#L616-L622) 
Make sure to install sacremoses with `pip install sacremoses`.  If you do no have `tokenizers` and a `tokenizer.json` file exists on the hub, you just need to install `tokenizers` as well



Hi Arthur, 
 
We are using '4.31.0' transformers. 

Is there any possibility of adding this library to Transformer package?

No, Transformers does not even have PyTorch as dependency. We keep the core dependency of the library minimal, we can't have every user required to install all the dependencies of the 200+ models.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Support for diarization in Whisper
Author: ldenoue
Pull request: no

### Feature request

In addition to word level timestamps, it would be very useful to get speaker turns like ggml whisper [recently added](https://github.com/ggerganov/whisper.cpp/commit/70e6fcd78b677b5a126010a4ad8e111c274af3f7).


### Motivation

I often transcribe podcasts or meeting recordings, which often involve more than one person speaking, so having the speaker turns would help tremendously.

### Your contribution

I could test the output using Whisper from GGML.

Comments:
cc @sanchit-gandhi and @xenova FYI 

@ldenoue As mentioned [on twitter](https://twitter.com/xenovacom/status/1687763550482096128), I'll add diarization to transformers.js once ready here 😇  (so, the title should be updated to "Support for diarization in Whisper")

Hey @ldenoue - I dug a bit deeper into this and it looks like the directionality for development was actually:

```
HF Whisper -> fine-tune with speaker turns ->  export to tinydiarize -> integrate with Whisper cpp
```

See this README for details: https://github.com/akashmjn/tinydiarize/tree/main#more-info

This means that we just need the author to push the Transformers weights to the Hub. After this, we can call `.from_pretrained` as usual to get the fine-tuned diarized model. There should be minimal code changes to get it working after this

I've opened a request for the weights here: https://github.com/akashmjn/tinydiarize/issues/15

Feel free to give it some weighting! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
download "wiki_dpr" dataset but not embeddings
Author: MaskXman
Pull request: no

### System Info

变压器-4.31.0
OS:macOS Ventura13.4

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

from transformers import RagTokenizer, RagRetriever, RagTokenForGeneration
from datasets import load_dataset

dataset = load_dataset("csv", data_files="/Volumes/WD_BLACK/datasets/psgs_w100.tsv", delimiter="\t")
data_tarin = dataset["train"]
data_tarin.save_to_disk("/Volumes/WD_BLACK/datasets/")
print(data_tarin)

tokenizer = RagTokenizer.from_pretrained("facebook/rag-token-nq")
retriever = RagRetriever.from_pretrained("facebook/rag-token-nq", index_name="exact",indexed_dataset=data_tarin)
model = RagTokenForGeneration.from_pretrained("facebook/rag-token-nq", retriever=retriever)

input_dict = tokenizer.prepare_seq2seq_batch("who holds the record in 100m freestyle", return_tensors="pt")

generated = model.generate(input_ids=input_dict["input_ids"])
print(tokenizer.batch_decode(generated, skip_special_tokens=True)[0])

### Expected behavior

我 want to see the correct dataset but not , I get the datasets without "embeddings" col
what should I do so that I can get the embeddings

Comments:
Please use the forums for such questions, as we keep issues for bugs in the library and feature requests only.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
reformat: change the eval_dataset * num_device code
Author: pphuc25
Pull request: yes

Hi,
In this code, it appears that there might be some confusion regarding the assign of per_device_eval_batch_size to another variable and then multiplying it by jax.device_count(). However, there is a discrepancy between the code styles used for training and evaluation, which makes it less clear. To improve clarity and align with the coding style used for train_batch_size, I have simplified eval_batch_size to a single line for a cleaner and more consistent coding style.
I would like cc @sanchit-gandhi to review my PR.

Comments: