noamaanMulla-03/transformers-issues huggingface dataset:
Update `Wav2Vec2ProcessorWithLM` doc example
Author: ydshieh
Pull request: yes

# What does this PR do?

The doctest for this doc example starts to fail. Even I roll back to commits a few days ago, it still fails. I think it's a dataset issue, so just updating the expected values.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I would rather make sure that the dataset was indeed changed! But otherwise ok for this

> I would rather make sure that the dataset was indeed changed! But otherwise ok for this

It's caused by the new release of `datasets==2.11.0` yesterday.

> It's caused by the new release of datasets==2.11.0 yesterday.

@ydshieh I'm assuming yes, but if we pin to an old version of datasets, do we have the old values? 

> > It's caused by the new release of datasets==2.11.0 yesterday.
> 
> @ydshieh I'm assuming yes, but if we pin to an old version of datasets, do we have the old values?

Yes, that's why I see the issue is coming from the datasets version :-)
[Pix2Struct] Add support to resize embeddings
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds `resize_token_embeddings` support for Pix2Struct. This was required when I fine-tuned Pix2Struct on a key-value pair dataset (the one from [this Donut notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Donut/CORD/Fine_tune_Donut_on_a_custom_dataset_(CORD)_with_PyTorch_Lightning.ipynb)). It oftentimes helps to add additional special tokens to the language decoder.

However, I noticed `tie_word_embeddings` is set to `True` in both the general config of Pix2Struct (`Pix2StructConfig`) as well as its text config (`Pix2StructTextConfig`). Printing out the weights of the decoder's embedding layer and its language modeling head seems to reveal weights aren't tied:

```
from transformers import Pix2StructForConditionalGeneration

model = Pix2StructForConditionalGeneration.from_pretrained("google/pix2struct-base")

print(model.decoder.embed_tokens.weight)
print(model.decoder.lm_head.weight)
```

So before merging this PR, we probably need to update the `tie_word_embeddings` attribute in the config of the models. Cause when you would load the model with this branch, it would break.

Currently you have to do:

```
from transformers import Pix2StructConfig, Pix2StructForConditionalGeneration

config = Pix2StructConfig(text_config={"tie_word_embeddings": False}, tie_word_embeddings=False)

model = Pix2StructForConditionalGeneration.from_pretrained("google/pix2struct-textcaps-base", config=config)
```

to make it work.

The PR also fixes some typos in configuration_pix2struct.py.

cc @younesbelkada 



Comments:
_The documentation is not available anymore as the PR was closed or merged._

@NielsRogge could you share a notebook on finetuning in this dataset?
Errror even if offload_dir is provided
Author: toncho11
Pull request: no

### System Info

Windows 10
16 GB RAM
4GB NVIDIA

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

You need CUDA enabled system.
You also need a version of transofrmers that has no other bugs and one that has the LlamaTokenizer available.
I am using transformers 4.28.0-dev, but not the latest as it has bugs.
Executing the following code: 
```
import torch
from peft import PeftModel
import transformers
import os, time
import tempfile

assert ("LlamaTokenizer" in transformers._import_structure["models.llama"]), "LLaMA is now in HuggingFace's main branch.\nPlease reinstall it: pip uninstall transformers && pip install git+https://github.com/huggingface/transformers.git"
from transformers import LlamaTokenizer, LlamaForCausalLM, GenerationConfig

tokenizer = LlamaTokenizer.from_pretrained("decapoda-research/llama-7b-hf")

BASE_MODEL = "decapoda-research/llama-7b-hf"
LORA_WEIGHTS = "tloen/alpaca-lora-7b"

force_cpu = False

if torch.cuda.is_available() and not force_cpu:
    device = "cuda"
    print("Video memory available:", torch.cuda.get_device_properties(0).total_memory / 1024 / 1024, "MBs")
else:
    device = "cpu"
print("Compute device is:", device)

try:
    if torch.backends.mps.is_available():
        device = "mps"
except:
    pass

print("Loading model with selected weights ...")

if device == "cuda":
    
    print("model on cuda")
    
    model = LlamaForCausalLM.from_pretrained(
        BASE_MODEL,
        load_in_8bit=False,
        torch_dtype=torch.float16,
        device_map="auto",
        offload_folder="offload", #required on GPU with not enough memory
    )
    
    model = PeftModel.from_pretrained(
        model, 
        LORA_WEIGHTS, 
        torch_dtype=torch.float16,
        device_map="auto",
        offload_folder="offload", #required on GPU with not enough memory
        
    )
```

Gives the error:

`ValueError: We need an `offload_dir` to dispatch this model according to this `device_map`, the following `

The offload folder is specified in my code and it works the first time, but not the second. I am forced to hard code the folder like this: `offload_dir = "offload"` to big_modeling.py at line 341 in order to avoid the error.

### Expected behavior

No error.

Comments:
Actually this is in accelerate. I need to move it there.
"LlamaTokenizer" in transformers._import_structure["models.llama"]                     │ │     9 ), "LLaMA is now in HuggingFace's main branch.\nPlease reinstall it: pip uninstall trans   │ │    10 from transformers import LlamaTokenizer, LlamaForCausalLM, GenerationConfig
Author: lucasjinreal
Pull request: no

### System Info

transformers main does have LlamaTokenizer


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

 pip uninstall transformers && pip install git+https://github.com/huggingface/transformers.git

### Expected behavior

Have it

Comments:
Solved.

> Solved.

How?
Could you tell us more?

> > Solved.
> 
> How? Could you tell us more?

pip install sentencepiece && pip install git+https://github.com/huggingface/transformers.git

Related: #22222 ?

I do not understand. It was not working with the latest version, but now it seems to work. Maybe somebody updated something very recently. Still the name change is confusing: LlamaTokenizer -> LLaMATokenizer
Update Neptune callback docstring
Author: normandy7
Pull request: yes

# What does this PR do?

Updates to the `NeptuneCallback` docstring:

- Update links to Neptune docs following migration
- Formatting

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@sgugger 

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer: @stas00, Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for your PR! For the formatting, could you apply `make style` on your branch (after a `pip install -e .["quality"]`) so that it's auto-fixed? In particular multi-line argument descriptions need to be all in the indented blocks.
ValueError: Reference at 'refs/heads/master' does not exist
Author: deema-A
Pull request: no

Hi, 
in the RAG example. 
I got the error (ValueError: Reference at 'refs/heads/master' does not exist)
in 
opt/anaconda3/envs/chatbotper/lib/python3.7/site-packages/git/refs/symbolic.py", line 184, in_get_ref_info_helper
    raise ValueError("Reference at %r does not exist" % ref_path)

after running: 
python examples/research_projects/rag/finetune_rag.py \
    --data_dir data_dir \
    --output_dir output_dir \
    --model_name_or_path facebook/rag-sequence-nq \
    --model_type rag_sequence \
    --fp16 \
    --gpus 8
    --index_name custom
    --passages_path path/to/data/my_knowledge_dataset
    --index_path path/to/my_knowledge_dataset_hnsw_index.faiss

any advice? thanx

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Same question from training yolov5, it seems that there are no effective solutions...
Fixed a bug in remove_handler function
Author: bahgat-ahmed
Pull request: yes

Fixed the bug mentioned in this issue: #21506

And replaced `Assertion`s with `ValueError`s only in that same function that contained the bug.

@LysandreJik

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_21718). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Evaluate QA models using run_qa.py script
Author: Yulong-W
Pull request: no


Comments:
The same command given in the README without the training prompts works perfectly. Just tried:
```
python examples/pytorch/question-answering/run_qa.py \
  --model_name_or_path deepset/deberta-v3-base-squad2 \
  --dataset_name squad \
  --do_eval \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir ~/tmp/debug_squad/
```
And it proceeds without error to an F1 of 92.28 and accuracy of 85% for exact matches.

You will need to adapt the example (which is just that, an example) to the format of your own data.
ValueError: invalid literal for int() with base 10: ''
Author: OrigamiDream
Pull request: no

### System Info

- `transformers` version: 4.27.3
- Platform: macOS-13.2.1-x86_64-i386-64bit
- Python version: 3.8.16
- Huggingface_hub version: 0.13.3
- PyTorch version (GPU?): 1.12.1 (False)
- Tensorflow version (GPU?): 2.11.1 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: True (via Horovod)

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

This might complex, but this happens very occasionally.
```
[1,0]<stderr>:Traceback (most recent call last): 
[1,0]<stderr>:  File "run_training.py", line 12, in <module> 
[1,0]<stderr>:    from proj.modeling import Model1, Model2
[1,0]<stderr>:  File "/model_proj/proj/modeling.py", line 6, in <module> 
[1,0]<stderr>:    from transformers import BertConfig, TFBertMainLayer 
[1,0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/transformers/__init__.py", line 26, in <module> 
[1,0]<stderr>:    from . import dependency_versions_check 
[1,0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/transformers/dependency_versions_check.py", line 17, in <module> 
[1,0]<stderr>:    from .utils.versions import require_version, require_version_core 
[1,0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/transformers/utils/__init__.py", line 56, in <module> 
[1,0]<stderr>:    from .hub import ( 
[1,0]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/transformers/utils/hub.py", line 1085, in <module> 
[1,0]<stderr>:    cache_version = int(f.read()) 
[1,0]<stderr>:ValueError: invalid literal for int() with base 10: '' 
```

1. Just import Transformers, and the error happens, randomly.

Retrying import fixes the issue, so I had ignored this for a while.
However, this had broken some of my infrastructures yesterday.

This must be fixed.

### Expected behavior

It must be imported without errors.

Comments:
Thanks for reporting! The PR above should fix this.

Closing issue since the PR has been merged into main branch :)
[WIP] pass kwargs to config
Author: cceyda
Pull request: yes

# What does this PR do?
Fixes https://github.com/huggingface/transformers/issues/21757


## Before submitting

- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger @Narsil
PretrainedConfig related

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_21758). All of your documentation changes will be reflected on that endpoint.

👍  Anyway, wasn't expecting changing something as fundamental as `.from_pretrained` to be reasonable or easy 😅  

@sgugger 
While working on this I realized a couple of things. Will make separate PRs for them if need be
- pruned_heads key values should be checked to be type int before casting + error message. there was also a test that used "a" as pruned_head but wasn't failing. will look into why later.
- Some models' configs use `initializer_range` some use `init_std`. 
For example while `FlaubertConfig` doesn't have `initializer_range` but  tests in FlaubertModelTester pass `initializer_range` and not `init_std`... These keys don't seem to defined int the `attribute_map` either. So should probably look into those.

Having fun figuring out how `from_pretrained` magic works

The pruned head fix is a welcome one. As I've said before (and as you can see from all the failing tests), you cannot change the logic inside the pretrained config like this without breaking many things in the library.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Slow decoding with many special tokens in vocabulary
Author: samsontmr
Pull request: no

### System Info

present across multiple versions

### Who can help?

@younesbelkada 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import T5Tokenizer
from time import time
from random import randint

t1 = T5Tokenizer.from_pretrained('t5-base')
t2 = T5Tokenizer.from_pretrained('t5-base', extra_ids=2000)
to_decode = [randint(0, 32000) for i in range(10000)]

start = time()
t1.decode(to_decode)
print("few special tokens:", time() - start)

start = time()
t2.decode(to_decode)
print("many special tokens:", time() - start)
```

### Expected behavior

The slowdown should not be so drastic. The cause is an inefficient implementation of [`all_special_ids`](https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils_base.py#L1293) and [`all_special_tokens`](https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils_base.py#L1267). Additionally, generating them on the fly incurs a large overhead since this attribute is queried for every id to be decoded ([here](https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils.py#L907) and [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/t5/tokenization_t5.py#L324)).

Comments:
Slow tokenizers are... slow. That's why we wrote the tokenizers library ;-) Why not use `T5TokenzierFast` which doesn't have the same problem?

`T5TokenzierFast` does not have byte-fallback + why artificially handicap the slow tokenizer if it could be more efficient (using sets instead of lists and computing the attribute only when it's updated)?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
LLaMA tokenizer seems to be broken
Author: SupreethRao99
Pull request: no

### System Info

- huggingface_hub version: 0.13.3
- Platform: Linux-5.19.0-32-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Token path ?: /root/.cache/huggingface/token
- Has saved token ?: True
- Who am I ?: supreethrao
- Configured git credential helpers: 
- FastAI: N/A
- Tensorflow: N/A
- Torch: 1.14.0a0+44dac51
- Jinja2: 3.1.2
- Graphviz: N/A
- Pydot: N/A
- Pillow: 9.2.0
- hf_transfer: N/A
- ENDPOINT: https://huggingface.co
- HUGGINGFACE_HUB_CACHE: /root/.cache/huggingface/hub
- HUGGINGFACE_ASSETS_CACHE: /root/.cache/huggingface/assets
- HF_TOKEN_PATH: /root/.cache/huggingface/token
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False

### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I converted the LLaMA weights to the HuggingFace format from the script in the documentation 
```
python src/transformers/models/llama/convert_llama_weights_to_hf.py \
    --input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path
```

When I try and load the tokenizer as follows
```
>>> from transformers import LlamaTokenizer
>>> tokenizer = LlamaTokenizer.from_pretrained('path_to_converted_llama_model')
```
I get the following error
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/transformers/tokenization_utils_base.py", line 1811, in from_pretrained
    return cls._from_pretrained(
  File "/usr/local/lib/python3.8/dist-packages/transformers/tokenization_utils_base.py", line 1965, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/llama/tokenization_llama.py", line 78, in __init__
    self.sp_model.Load(vocab_file)
  File "/usr/local/lib/python3.8/dist-packages/sentencepiece/__init__.py", line 905, in Load
    return self.LoadFromFile(model_file)
  File "/usr/local/lib/python3.8/dist-packages/sentencepiece/__init__.py", line 310, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
```
I have the following versions of the library
```
transformers            4.28.0.dev0 (installed from source though pip install git+https://github.com/huggingface/transformers.git)
sentencepiece           0.1.97
```


### Expected behavior

The tokenizer should get loaded be able function properly without the aformentioned errors

### Edit
The tokenizer.model file was corrupted which caused this issue, once that was fixed, conversion and tokenization works. 
closing this issue now

Comments:
Update convert_llama_weights_to_hf.py
Author: Ricardokevins
Pull request: yes

# What does this PR do?

Fix problem mentioned in https://github.com/huggingface/transformers/issues/22287

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts
@ArthurZucker




Comments:
_The documentation is not available anymore as the PR was closed or merged._

Well this was bound to disappear with #22402 😅