noamaanMulla-03/transformers-issues huggingface dataset:
Going above version 4.21.3 gives UnicodeDecodeError
Author: emidio90
Pull request: no

### System Info

- `transformers` version: 4.27.2
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.10.6
- Huggingface_hub version: 0.13.3
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. update stable diffusion webui to a version that requires Transformers 4.22.0 or above
2. start stable diffusion
3. when loading the model, this error appears
![Screenshot 2023-04-08 232922](https://user-images.githubusercontent.com/15109972/230743894-dbc3598c-15b9-4222-ad80-51cef9eb4fb8.png)


### Expected behavior

load the model normally like it does for version 4.21.3 and below

Comments:
cc @amyeroberts and @younesbelkada 

Hi @emidio90, thanks for raising this issue. 

Could you share some more information about how to reproduce this error? In particular, is this the [Stable Diffusion WebUI](https://github.com/AUTOMATIC1111/stable-diffusion-webui) repo being referred to? What model and settings are being used? And which task is being run? 

Hello @amyeroberts , I confirm that is the repo i'm referring to.
This happens with any model, when starting WebUI locally on my pc by clicking on webui-user.bat. The console arguments are only --xformers.
I'm sorry i don't know how to find the specific task being run, but I suppose it's the model loading that happens when launching webui.
This started to happen when I updated to a commit that changed the required version of Transformers to 4.25. So now to make SD work I have to manually change the requirements_version.txt transformers==4.21.3

@emidio90 Great, thanks for the additional info! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
For xla tensors, use an alternative way to get a unique id
Author: qihqi
Pull request: yes


# What does this PR do?

XLA tensors don't have storage and attempting to get it's storage will get

RuntimeError: Attempted to access the data pointer on an
invalid python storage.

Repro:
```
from transformers import pytorch_utils
import torch_xla.core.xla_model as xm
device = xm.xla_device()
a = torch.ones((10,10)).to(device)
pytorch_utils.id_tensor_storage(a)  # raises RuntimeError
```
With this patch it would print out
(device(type='xla', index=0), 1, 400).

Comments:
Hi @muellerzr moved https://github.com/huggingface/safetensors/pull/349 to here

LMK if I should add a test (and in which file) I am not sure if the CI machines has torch_xla installed  so `import torch_xla` can succeed...

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25802). All of your documentation changes will be reflected on that endpoint.

@amyeroberts this stems from https://github.com/huggingface/safetensors/pull/349, where we specified that it should be done in `transformers` and `accelerate` separately.
Saving with trainer deepspeed zero3 missing config.json and tokenizer files.
Author: zjjMaiMai
Pull request: no

trainer will not save tokenizer and config.json when training in deepspeed-**zero3** with `stage3_gather_16bit_weights_on_model_save=False`.

line 2776 will `raise ValueError`, so line 2778 `self._save` never run to save tokenizer and other stuff. is this expected behavior?

https://github.com/huggingface/transformers/blob/d4bd33cc9f11ca48635e54983d75249c78d72e2a/src/transformers/trainer.py#L2771-L2784

_Originally posted by @zjjMaiMai in https://github.com/huggingface/transformers/issues/24728#issuecomment-1669067573_
            

Comments:
cc @pacman100 

Hello @zjjMaiMai, Thank you for the details. This shouldn't be the behaviour and I'll be working on fixing this.
Can't import Blip2Processor
Author: tanayvarshney
Pull request: no

### System Info

I was trying to follow [this tutorial](https://huggingface.co/docs/transformers/main/en/model_doc/blip-2#transformers.Blip2Model.forward.example) and ran into the following issue:
```
ImportError: cannot import name 'Blip2Processor' from 'transformers' (/usr/local/lib/python3.8/dist-packages/transformers/__init__.py)
```

version: '4.26.1'

@sgugger @ArthurZucker @amyeroberts (there is no PIC for multimodal models so tagging both PICs)

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Just run the following example: https://huggingface.co/docs/transformers/main/en/model_doc/blip-2#transformers.Blip2Model.forward.example

### Expected behavior

Should import the preprocessor

Comments:
You need to install transformers from source to get access to BLIP-2:
```
pip install git+https://github.com/huggingface/transformers
```

@sgugger thank you! Just one minor feedback since you are the docs PIC, if a note modal or just a highlighted note could be added informing if a module hasn't been added to the stable release would be helpful. (If it is already there, I might have missed it, apologies.)

Note that it's not in the [stable documentation](https://huggingface.co/docs/transformers/index) (which is what is viewed by default) only the [main documentation](https://huggingface.co/docs/transformers/main/en/index). Did you get on the page via a search engine maybe and did not realize you were not on the documentation of the latest release?

> Note that it's not in the [stable documentation](https://huggingface.co/docs/transformers/index) (which is what is viewed by default) only the [main documentation](https://huggingface.co/docs/transformers/main/en/index). Did you get on the page via a search engine maybe and did not realize you were not on the documentation of the latest release?

I am not the author but this is exactly what happened to me - I did not see at all there's a dropdown for versions so I assumed BLIP2 is just available.

🤓 

Please restart your notebook, and re-install using below command.

`pip install git+https://github.com/huggingface/transformers`
POET
Author: rajveer43
Pull request: no
Labels: name: New model

## Train BERT and other large models on smartphones

### Model description

`POET` enables the training of state-of-the-art memory-hungry ML models on smartphones and other edge devices. POET (Private Optimal Energy Training) exploits the twin techniques of integrated tensor rematerialization, and paging-in/out of secondary storage (as detailed in our paper at ICML 2022) to optimize models for training with limited memory. POET's Mixed Integer Linear Formulation (MILP) ensures the solutions are provably optimal! approach enables training significantly larger models on embedded devices while reducing energy consumption while not modifying mathematical correctness of backpropagation. We demonstrate that it is possible to fine-tune both `ResNet-18` and `BERT` within the memory constraints of a Cortex-M class embedded device while outperforming current edge training methods in energy efficiency.



### Open source status

- [X] The model implementation is available
- [ ] The model weights are available

### Provide useful links for the implementation

[Implementation](https://github.com/ShishirPatil/poet)
[Paper](https://arxiv.org/abs/2207.07697)
[Author's Website](https://shishirpatil.github.io/poet/)

Comments:
Hi @rajveer43, thanks for opening this issue! 

transformers is a library of models, rather than optimizers. This might be better placed in the optimum library - WDYT @younesbelkada? 

Hi @rajveer43 ! 
Thanks for the request, indeed, as pointed out by amy, this sounds more in the scope of optimum library! 
However I am not sure how this should be addressed as this scenario is pretty unique, if I am not mistaken this would be the first time having an optimizer in optimum. 
Curious to hear @fxmarty @michaelbenayoun @JingyaHuang 's thoughts here !
Thanks!
llama fast tokenizer: FileNotFound error when saving model checkpoint and self.vocab_file does not exist
Author: ZhangShiyue
Pull request: no

### System Info

transformers==4.31.0
torch==2.0.1

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Traceback (most recent call last):
....
  File "python3.9/site-packages/transformers/trainer.py", line 1539, in train
    return inner_training_loop(
  File "python3.9/site-packages/transformers/trainer.py", line 1916, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "python3.9/site-packages/transformers/trainer.py", line 2237, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "python3.9/site-packages/transformers/trainer.py", line 2294, in _save_checkpoint
    self.save_model(output_dir, _internal_call=True)
  File "python3.9/site-packages/transformers/trainer.py", line 2749, in save_model
    self._save(output_dir, state_dict=state_dict)
  File "python3.9/site-packages/transformers/trainer.py", line 2832, in _save
    self.tokenizer.save_pretrained(output_dir)
  File "python3.9/site-packages/transformers/tokenization_utils_base.py", line 2221, in save_pretrained
    save_files = self._save_pretrained(
  File "python3.9/site-packages/transformers/tokenization_utils_fast.py", line 595, in _save_pretrained
    vocab_files = self.save_vocabulary(save_directory, filename_prefix=filename_prefix)
  File "python3.9/site-packages/transformers/models/llama/tokenization_llama_fast.py", line 186, in save_vocabulary
    copyfile(self.vocab_file, out_vocab_file)
  File "/opt/bb/lib/python3.9/shutil.py", line 264, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: './model/tokenizer.model'

### Expected behavior

When I finetune llama, it threw out this error when saving the first checkpoint because the original model directory was deleted. 

I noticed that in https://github.com/huggingface/transformers/blob/ef1534252f76231b4a6403c71866d4376e35292d/src/transformers/models/llama/tokenization_llama.py#L281, it checks whether self.vocab_file exists. 

Could this be added to tokenization_llama_fast.py too?

Comments:
Sure, the problem is that in fast we cannot recover the content of `vocab_file` if the repo was deleted. We can produce a warning however, mentioning that you won't be able to initialize a slow tokenizer. Opening a PR to fix this! Thanks for reporting

Thanks a lot! Does it mean a fast tokenizer can still be initialized if vocab_file does not exist?

It depends, if you have a `tokenizer.json` file then yes, if not, then you cannot convert the slow tokenizer if the `vocab_file` (which in this case is the sentencepiece model) was deleted no?
[`TokenizerFast`] `can_save_slow_tokenizer` as a property for when `vocab_file`'s folder was removed
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fixes #25602, making `can_save_slow` a property rather than an attribute as we need to check if the vocab_file still exists! 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[`CodeLlama`] Fix CI 
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Nit for `can_save_slow_tokenizer` for Llama cc @amyeroberts. Merging with main made CodeLlama fail as the property was not defined as such. See #256626

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Save image_processor while saving pipeline (ImageSegmentationPipeline)
Author: raghavanone
Pull request: yes

# What does this PR do?

Fixes #25804 

When saving the pipeline, we had missed saving image_processor which was causing error when loading pipeline from disk. This PR fixes this bug.

@ArthurZucker 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25884). All of your documentation changes will be reflected on that endpoint.

> Nice. Can you add a little bit more details on the PR description? 😉

Done

FYI @amyeroberts !
Reduce CI output
Author: ydshieh
Pull request: yes

# What does this PR do?

The use of `-s` will show a lot of outputs: the `torch_job` has more than 1.6M lines in the outputs, which makes the loading very long and hard to see the final failed test names at the end.

Let's not use this flag. If this affects other core maintainer's workflow, let's see what to do alternatively.

Comments:
@ArthurZucker Let me merge this (despite not urgent), and let me know if you have some opinion of the new output shown on CircleCI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25876). All of your documentation changes will be reflected on that endpoint.

<img width="626" alt="Screenshot 2023-08-31 at 10 48 57" src="https://github.com/huggingface/transformers/assets/48595927/92a07529-5aff-481a-9098-febf2889baa8">
It's a lot better IMO

And probably faster since rendering 

Yes! Too much info -> we never look the  trackback in that layout but in the artifact 😅
[`InstructBlip`] FINAL Fix instructblip test
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes the current failing instructblip slow test

This commit: https://github.com/huggingface/transformers/commit/2230d149f0dc4de05ea7a9a1c5295b6d80e495cd was responsible of the test failure however the commit above sillently fixed an issue.

The PR #25105 introduced the correct way of quantizing composable models (blip2, instructblip) and models on the Hub. Before that commit the lm_head of the language model was converted in 8bit, which in fact is wrong. The lm_head should always stay in fp32 for numerical stability and also for consistency with other causal LM models in the library (we keep all lm_head in fp32). Therefore this PR fixes the expected logits and generations

Tested the fix in the latest docker image `huggingface/transformers-all-latest-gpu` on a 2xNVIDIA T4 and the test pass 

cc @ydshieh @amyeroberts 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

So #25105 fixed (among other things) that `lm_head was in 8bit`, right?

Confirmed it passes when running the single test, but failed when running the 2 integration tests together.

@amyeroberts I think we can merge, and I will try to figure out why this strange situation happens. WDYT?

> So https://github.com/huggingface/transformers/pull/25105 fixed (among other things) that lm_head was in 8bit, right?

Yes, specifically for models that has submodules that are causalLM models, mainly instructblip and blip2
GPTQ Quantization via `from_pretrained`: why enforcing `fp16`?
Author: HanGuo97
Pull request: no

### Feature request

Hi, I noticed in the following line that model has to be in `fp16` format before GPTQ quantization. I'm curious whether this condition can be dropped?
https://github.com/huggingface/transformers/blob/ccb92be23def445f2afdea94c31286f84b89eb5b/src/transformers/modeling_utils.py#L2504


### Motivation

My use case runs into troubles with `fp16` but works with `bf16`, and I noticed that if I simply remove this line and keep `torch_dtype=None` everything runs fine.

### Your contribution

NA

Comments:
cc @younesbelkada 

Hi ! 
That might be a copypasta from previous bnb integration but not sure, we should probably override it to `torch.float16` only if `torch_dtype=None`, @SunMarc what do you think? Maybe there is something I have overlooked about GPTQ 

Why override it if `torch_dtype=None`? I think `fp32` runs just fine too?

if we set it to float32 by default it will create a lot of overhead for non-linear modules being in fp32 (such as embedding layer) making it not possible to fit some models on Google colab for example, therefore for bnb we set them to half-precision with a logger.info explaining what is happening under the hood. 
(for bnb) you can always cast the non-linear modules in fp32 by sending `torch_dtype=torch.float32`

Understood -- thanks for the explanation!

So just to confirm, there are no correctness issues with using `torch.float32`, it’s just that using `fp16` instead can result in better efficiency for certain workloads?

If that’s the case, would it be more effective to add a warning when `torch_dtype=None` and suggest using `fp16` for better efficiency? Personally, I prefer having fewer overrides, but I’m open to either approach.

Hi @HanGuo97 , the backend in auto_gptq library always used` torch_dtype = torch.float16` by default and I ran into a couple of issues with `torch.dtype = torch.float32` in the past most probably due to how the kernels were implemented. So this is why i hardcoded to `torch.float16`. But I guess that if it works for you, I will do as you suggested ! 

Interesting, thanks for the clarification!

I briefly looked into the `auto_gptq` library, and I think they have a [different code](https://github.com/PanQiWei/AutoGPTQ/blob/main/auto_gptq/nn_modules/qlinear/qlinear_cuda_old.py#L203) path depending on whether the data is in `fp16` or not.

Yeah, I must have forgotten to deactivate `use_cuda_fp16` as it is enabled by default ;)

Oh yes you are right, I missed this :)

(In hindsight, it's a bit odd they set this to True by default when it clearly depends on the model.)

Edit: `optimum` will detect the proper flag [here](https://github.com/huggingface/optimum/blob/main/optimum/gptq/quantizer.py#L281)

Thanks again for looking into that !
Fast and normal tokenizer generate different output when handling consecutive spaces.
Author: torshie
Pull request: no

### System Info

Transformer version: 4.29.2

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Code to reproduce:
```python
import transformers

fast = transformers.AutoTokenizer.from_pretrained('openlm-research/open_llama_7b', use_fast=True, cache_dir='hf_cache')
normal = transformers.AutoTokenizer.from_pretrained('openlm-research/open_llama_7b', use_fast=False, cache_dir='hf_cache')

print(fast.__class__, fast('a    b'))
print(normal.__class__, normal('a    b'))
```

output:
```text
<class 'transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast'> {'input_ids': [1, 260, 31822, 31822, 31822, 284], 'token_type_ids': [0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1]}
<class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'> {'input_ids': [1, 260, 284], 'attention_mask': [1, 1, 1]}
```

### Expected behavior

Same output

Comments:
cc @ArthurZucker 

This seems to be pretty close to #24918. 
I don't know how they converted the fast tokenizer but it seems wrong. I suggest you open an issue at https://hf.co/openlm-research/open_llama_7b/discussions, as if you use the `huggyllama/llama-7b` model, you will not have this issue: 

```python 
import transformers

>>> fast = transformers.AutoTokenizer.from_pretrained('openlm-research/open_llama_7b', use_fast=True, cache_dir='hf_cache')
>>> normal = transformers.AutoTokenizer.from_pretrained('openlm-research/open_llama_7b', use_fast=False, cache_dir='hf_cache')

>>> print(fast.__class__, fast('a    b'))
<class 'transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast'> {'input_ids': [1, 263, 1678, 289], 'attention_mask': [1, 1, 1, 1]}
>>> print(normal.__class__, normal('a    b'))
<class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'> {'input_ids': [1, 263, 1678, 289], 'attention_mask': [1, 1, 1, 1]}
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Falcon: Add RoPE scaling
Author: gante
Pull request: yes

# What does this PR do?

In the same spirit as #24653, adds RoPE scaling to Falcon. It also borrows a few changes from #25740, to allow for `codellama`-style scaling.

In addition to the changes above, it also adds the `max_position_embeddings` to the config attributes, needed for one of the scaling strategies.


__________________________________________

Python script to validate these changes: https://pastebin.com/SJmUpDU1
Before this PR 👉 outputs gibberish
After this PR 👉 recognizes that the super large prompt is about llama 2

Comments:
_The documentation is not available anymore as the PR was closed or merged._

`t` and `einsum` PR comments 👉  as discussed offline, this will be fixed in a follow-up PR

I've also tested this PR against _the thing I wanted to test_, it is working correctly with and without RoPE scaling!

Merging :)
How to re-tokenize the training set in each epoch?
Author: tic-top
Pull request: no

I have a special tokenizer which can tokenize the sentence based on some propability distribution.
For example, 'I like green apple' ->'[I],[like],[green],[apple]'(30%) or '[I],[like],[green apple]' (70%).
Now in the training part, I want the Trainer can retokenize the dataset in each epoch. How can I do so?

Comments:
Please ask this kind of questions on [the forum ](https://discuss.huggingface.co/)
Support for the custom kernel for Mask2Former and OneFormer
Author: function2-llx
Pull request: no
Labels: name: Feature request

> And @shivalikasingh95 sure but let's add support for the custom kernel in a separate PR for Mask2Former and OneFormer. We can set it to False by default, and add a boolean attribute for those models.

_Originally posted by @NielsRogge in https://github.com/huggingface/transformers/pull/20993#discussion_r1062224249_

Hello, could I know if there's any plan to add support for the custom kernel for Mask2Former and OneFormer as mentioned there, thanks!

Comments: