noamaanMulla-03/transformers-issues huggingface dataset:
RuntimeError: Could not infer dtype of NoneType
Author: TekeshwarHirwani
Pull request: no

### System Info

I was using microsoft/biogpt and gpt2 model on old script of run_ner.py that is (https://github.com/huggingface/transformers/blob/main/examples/legacy/token-classification/run_ner.py ) since both have slow tokenizer and  they both encountered same error that is 

RuntimeError: Could not infer dtype of NoneType

I have used the same dataset given in the repo and changed model name that's it.

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. change the model name to microsoft/biogpt
2. Follow the instruction given on the repo

### Expected behavior

it should train successfully 

Comments:
Hi @TekeshwarHirwani 

The files under [transformers](https://github.com/huggingface/transformers/tree/main)/[examples](https://github.com/huggingface/transformers/tree/main/examples)/[legacy](https://github.com/huggingface/transformers/tree/main/examples/legacy) is no longer maintained: (from the README file)

> This folder contains examples which are not actively maintained (mostly contributed by the community).

> Using these examples together with a recent version of the library usually requires to make small (sometimes big) adaptations to get the scripts working.

You can use the files under [token-classification](https://github.com/huggingface/transformers/tree/main/examples/pytorch/token-classification) 🤗 

Thankyou for response, But the model I am using is microsoft/biogpt and it is having slow tokenizer, readme file of examples/tokenclassification/pytorch you have mentioned this :

Note: This script only works with models that have a fast tokenizer (backed by the 🤗 Tokenizers library) as it uses special features of those tokenizers. You can check if your favorite model has a fast tokenizer in [this table](https://huggingface.co/transformers/index.html#supported-frameworks), if it doesn't you can still use the old version of the script.


Hi @TekeshwarHirwani 

In this situation, you will have to dive into the error messages, set some breakpoints, investigate the values of some variables to figure out why there is a None value. And see if you can modify the `legacy` code to make it work.

You can  try [Hugging Face Forums](https://discuss.huggingface.co/) to see if someone else had the same issues and if there are already some approaches.

Thanks
How to use GPT2DoubleHeadsModel?
Author: h56cho
Pull request: no

Hello,

I have a question about the example shown in the GPT2DoubleHeadsModel documentation page:
https://huggingface.co/transformers/model_doc/gpt2.html#gpt2doubleheadsmodel

In the example, the input to the GPT2DoubleHeadsModel is simply a set of choices. But what if the multiple choice question that I want to process also includes a question text? so for example,

Bob likes candy; what does Bob like?
a. Bag
b. Burger
c. Candy
d. Pencil

In the example above, the question text would be " Bob likes candy; what does Bob like?"
and the choices will be the Bag, Burger, Candy and Pencil. How should I pre-process my multiple choice questions to be used with the GPT2DoubleHeadsModel?

For example, given that the token that will be used for the classification is "
" (this is the default eos token for the GPT2 models), would the following be fine?

```python
import torch
from transformers import GPT2Tokenizer, GPT2DoubleHeadsModel

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2DoubleHeadsModel.from_pretrained('gpt2')

choices = [ "Bob likes candy ; what does Bob like ?  Bag 
",
                   "Bob likes candy ; what does Bob like ?  Burger 
",
                   "Bob likes candy ; what does Bob like ?  Candy 
",
                  "Bob likes candy ; what does Bob like ?  Pencil 
"]

encoded_choices = [tokenizer.encode(s) for s in choices]
eos_token_location = [tokens.index(tokenizer.eos_token_id) for tokens in encoded_choices]

input_ids = torch.tensor(encoded_choices).unsqueeze(0) 
mc_token_ids = torch.tensor([eos_token_location]) 
outputs = model(input_ids, mc_token_ids=mc_token_ids)
lm_prediction_scores, mc_prediction_scores = outputs[:2]
```

Thank you,

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I see that the issue is closed? what is the answer?

We are trying to move such questions more and more to the forum because they get more traction there and the library's issues should primarily be used for "real" issues. 

It would be awesome if you guys could post the question on 
https://discuss.huggingface.co/

I meet the same question.
[`MT5`] Fix CONFIG_MAPPING issue leading it to load umt5 class
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Adresses #24662 and one of our CI test.
The issue stems from the `CONFIG_MAPPING`'s values used as keys to index over the auto mapping. 
There were two ways to fix this, either change our logic or just add a config.
For simplicity added a config. 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Can confirm that: 
```python 
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model = AutoModelForSeq2SeqLM.from_pretrained('google/mt5-small')
print(type(model))
<class 'transformers.models.mt5.modeling_mt5.MT5ForConditionalGeneration'>
```
is back to normal 😄
DeepMind Retro
Author: shamanez
Pull request: no
Labels: name: New model

# 🌟 New model addition

Basically a retrieval augmented model like RAG, but without expensive retriever end2end training



## Open source status

* [ ] the model implementation is available: Not available 
* [ ] the model weights are available:  Not available 
* [ ] who are the authors:(https://arxiv.org/pdf/2112.04426.pdf)


Comments:
Hi,

No weights are available at the moment. I'd suggest to take a look at this repo: https://github.com/lucidrains/RETRO-pytorch

Yeah, I saw it. Anyways I am thinking of combining Retro with RAG model. The results seem very promising also the model is very practical as recent work suggests end-to-end retriever training is very expensive and impractical.


I would love to get some help :)

> I would love to get some help :)

Hi! I am interested in this implementation and I can help for it!

@halixness  do you think we should modify the RAG implementation ? 

@shamanez @halixness @NielsRogge  I would like to participate in the implementation, please let me know

Hi All,

I think we can use the RAG pipeline.  But we need to implement the reader and its loss function. 

We might be able to use [this](https://github.com/lucidrains/RETRO-pytorch).

any suggestions? 

> I would love to get some help :)

I'd like to help as well!

yeah this model seems cool. feel like it's getting closer to true open source LLMs. anyone aware if there are similar open-source efforts as bloom to reproduce and publish this?

Hi @shamanez, very I'm interested in RETRO. How far along are you in an implementation?

I did some digging and apparently Nvidia is picking up this idea again. I found it via an [article](https://www.marktechpost.com/2023/04/22/this-ai-paper-from-nvidia-provides-the-recipe-to-reproduce-retro-up-to-9-5b-parameters-while-retrieving-a-text-corpus-with-330b-tokens/) from April '23. It links to their [GitHub repo](https://github.com/NVIDIA/Megatron-LM/tree/main/tools/retro) and the corresponding [paper](https://arxiv.org/pdf/2304.06762.pdf) by Nvidia. In this paper they describe how they reproduce RETRO including the generation of the pretraining corpus.
However, in the paper they mention that the implementation and the weights are not open-sourced, but just being able to regenerate the pretraining corpus would get us one step ahead.
Fix flaky `test_for_warning_if_padding_and_no_attention_mask`
Author: ydshieh
Pull request: yes

# What does this PR do?

This tests #24510. It fails a few times (on my PRs, and once in someone's PR), and now it fails on the latest daily CI.

This test is flaky because it test the functionality of `warn_if_padding_and_no_attention_mask` which uses `logger.warning_once(warn_string)` - this uses `@functools.lru_cache(None)`.

If there is any test that triggers this warning before `test_for_warning_if_padding_and_no_attention_mask`, the cache is produced, and we won't get the expected warning in `test_for_warning_if_padding_and_no_attention_mask` then it fails.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Breaking change in upcoming PyTorch version for weight norm and loading pretrained models
Author: MarktHart
Pull request: no

Probably to be fixed around here: https://github.com/huggingface/transformers/blob/bbf3090848cf0ceff98f9465691e9ecce63684a1/src/transformers/modeling_utils.py#L3016

See this issue on PyTorch:
https://github.com/pytorch/pytorch/issues/102999#issuecomment-1623975562



Comments:
Hi, what's your `transformers` version?

If you use a dev version with the commit of  this PR #https://github.com/huggingface/transformers/pull/24030 included, it should be fine. Let me know if not, thanks.
Pix2StructImageProcessor does not accept list of PIL Images
Author: LiJunnan1992
Pull request: no

### System Info

- `transformers` version: 4.30.2
- Platform: Linux-5.10.133+-x86_64-with-glibc2.10
- Python version: 3.8.13
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)

### Who can help?

@younesbelkada 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Pix2StructImageProcessor does not work if I pass in a list of PIL images as input. It works after I uncomment line 373-379: https://github.com/huggingface/transformers/blob/66fd3a8d626a32989f4569260db32785c6cbf42a/src/transformers/models/pix2struct/image_processing_pix2struct.py#L373

### Expected behavior

According to the documentation, Pix2StructImageProcessor should be able to process list of PIL images.

Comments:
Hi. Could you show us the full error log, please. Thanks.

cc @amyeroberts 

Hi @LiJunnan1992 

The script below seems to work on the main branch of transformers. Can you share a reproducible snippet? 🙏 Thanks!

```python
import requests
from PIL import Image
from transformers import Pix2StructProcessor

processor = Pix2StructProcessor.from_pretrained("google/pix2struct-textcaps-base")

url = "https://www.ilankelman.org/stopsigns/australia.jpg"
images = [Image.open(requests.get(url, stream=True).raw) for _ in range(4)]

inputs = processor(images, return_tensors="pt")
```

This script works as well:

```python
import requests
from PIL import Image
from transformers import Pix2StructProcessor, Pix2StructImageProcessor

processor = Pix2StructProcessor.from_pretrained("google/pix2struct-textcaps-base")

image_processor = Pix2StructImageProcessor()

url = "https://www.ilankelman.org/stopsigns/australia.jpg"
images = [Image.open(requests.get(url, stream=True).raw) for _ in range(4)]

_ = processor(images, return_tensors="pt")
_ = image_processor(images, return_tensors="pt")
```

The main branch indeed works without error. Closing this issue. Thanks!
implement unlimiformer into transformers
Author: chris-aeviator
Pull request: no

### Feature request

https://github.com/abertsch72/unlimiformer promises to support unlimited input length on any transformer based encoder/decoder model with sub-linear cost in time.

### Motivation

Context lengths are fairly limited

### Your contribution

Testing

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Bump

> Am 04.06.2023 um 17:01 schrieb github-actions[bot] ***@***.***>:
> 
> ﻿
> This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
> 
> Please note that issues that do not follow the contributing guidelines are likely to be ignored.
> 
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you authored the thread.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Time Series Transformer - Dynamic Categorical Features
Author: guyko81
Pull request: no
Labels: name: Feature request

### Feature request

I would like to have a Dynamic Categorical Feature Embedding option in TimeSeriesTransformerConfig 

### Motivation

I didn't see any option in the TimeSeriesTransformerConfig where I could define an embedding of a Dynamic Categorical Feature. I'm working with sales data and holiday is an important element of sales, so all of my models handle the holidays with a dynamic embedding. Is it the case in Time Series Transformer too, and I'm just missing something?

### Your contribution

Happy to help, but would need some guidance on how it's handled currently.

Comments:
cc @kashif 

@guyko81  yes sure! I would be happy to help you get this done. I never found a good example of dynamic categorical features, so if you have some sample example that would be really helpful. 

We can assume that the dataset has a key  e.g. 

```py
dynamic_static_categorical  = [ [0, 2, 555], [23, 5, 66], ... [33, 4, 54]]
```

where we have a list of categories for each time point where the len of this array will be the length of the target values array in the time dim.

Next we will need to specify the number of dynamic cat. features (3) in the example above and the cardinalities and dims of the corresponding features:

```
dynamic_cat_card = [50, 10, 1000]
dynamic_cat_dimns = [12, 16, 32]
```

Once we have that done on the config side we can just add a corresponding `nn.Embedding` and concat the outputs to the input vector. If you open a PR please CC me and then i can help out! 

Thank you!


@kashif I have created a pull request https://github.com/huggingface/transformers/pull/24712
Still need to test it first, but I wanted you to have a look
improve(llama): Faster apply_rotary_pos_emb
Author: fpgaminer
Pull request: yes

# What does this PR do?

Faster implementation for `apply_rotary_pos_emb` in `modeling_llama.py`.

Please see issue #22683 for code that verifies the correctness of the change.

NOTE: Not marking as fixing the above issue, as speed is still not as good as before.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@gante 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Should a similar patch be applied to GPT-NeoX?

@neggert I believe it can be added to GPT-NeoX too - very happy to review a PR if you'd like to add!
Model checkpoint twice as large when saved with safetensors
Author: lenbrocki
Pull request: no

### System Info

- `transformers` version: 4.30.2
- Platform: Linux-5.15.0-1037-gcp-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("facebook/opt-2.7b")
model.save_pretrained("opt_safetensor", safe_serialization=True)
```

The original pytorch_model.bin is 5.3GB and the new one is sharded:
model-00001-of-00002.safetensors 9.3GB
model-00002-of-00002.safetensors 601MB

### Expected behavior

Is that the expected behaviour? I would expect the weights to have roughly the same size when saved using safetensors.

Comments:
Hi @lenbrocki 

Could you make a self-complete code snippet in `Reproduction` section, please. Thank you.

I have updated the Reproduction section

Thanks @lenbrocki !

cc @Narsil 

This model is saved in `float16`.  `from_pretrained` will by default load it in `float32`.

`from_pretrained(..., torch_dtype=torch.float16)`

Should fix it.

Yes, that fixed it. Thanks!
Fixes all hidden states output in FlaxT5
Author: ztjhz
Pull request: yes

Fixes #23960

@sanchit-gandhi 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

need to fix the tests

Hmm okay so the PyTorch model is also missing this, so we'd need to update it here. Continuing the discussion in the issue!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Based on the discussion in the issue: https://github.com/huggingface/transformers/issues/23960#issuecomment-1579140627, we should probably not update the Flax and PyTorch T5 models since this would be a surprise breaking change to what is one of the most popular models in the lib. Feel free to make these changes locally @ztjhz if you need all the hidden states! Otherwise we can close this one for now

cc @amyeroberts
ValueError: DistilBertModel does not support gradient checkpointing.
Author: sachinya00
Pull request: no

How to enable the "gradient_checkpointing" for DistilBert model ? However, it's working fine for the Bert model, I've followed the steps given on this page to enable it
https://huggingface.co/docs/transformers/v4.18.0/en/performance

I've gone through the huggingface code of respective classes and found that the feature is present only for the Bert model and not the DistilBert. 
https://github.com/huggingface/transformers/blob/188a8bfcccc6b862fe7ccc2859d977c01dd98136/src/transformers/models/bert/modeling_bert.py#L593

https://github.com/huggingface/transformers/blob/188a8bfcccc6b862fe7ccc2859d977c01dd98136/src/transformers/models/distilbert/modeling_distilbert.py#L470



Comments:
Yes DistilBert does not support gradient checkpointing. DistilBERT is a small model, so that feature is not needed for it.

I want to run this model across large batch sizes to see how much I can benefit from this. Is there any way I can enable for this model as well using torch.utils.checkpoint.checkpoint, but not sure where to apply checkpointing for this.  

I want to try out DeepSpeed’s activation checkpointing but can't use this on above model as it requires to enable the "gradient_checkpointing" flag in the HF trainer.
I was going through the DeepSpeed details on the below page and it mentions that we've to enable the "gradient_checkpointing" flag in HF trainer to use this 

"**HF Transformers models don’t know anything about DeepSpeed’s activation checkpointing,** so if you try to enable that feature in the DeepSpeed config file, nothing will happen." 
(https://huggingface.co/docs/transformers/main_classes/deepspeed)

What code changes required I need to do to replace with the Deepspeed API or enable the  model.gradient_checkpointing_enable() for distilbert


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> Yes DistilBert does not support gradient checkpointing. DistilBERT is a small model, so that feature is not needed for it.

I don't think this feature is redundant if we want to train it with extremely large batch size. To my knowledge, minilm, which has fewer parameters than distilbert, supports gradient checkpointing.

Yeah, I agree jordan. When I tried to compare some transformer models, I could not train DistilBert because of the large batch size while i could train bert/roberta.