noamaanMulla-03/transformers-issues huggingface dataset:
I am getting below error message when loading XLM 17 Language pretrained model.
Author: Allricstan
Pull request: no

Some weights of XLMWithLMHeadModel were not initialized from the model checkpoint at xlm-mlm-17-1280 and are newly initialized: ['transformer.position_ids']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

Comments:
Can anyone help me how to use this model for language translation?


You can disregard the warning, it's issued wrongly (and it's been fixed on main).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add support for images embeddings as one of the **input parameters
Author: ramanova
Pull request: no

### Feature request

[Owl-ViT](https://huggingface.co/docs/transformers/model_doc/owlvit)
**Current flow** for inference for OWL ViT is to accept image and text, run it through a processor and then give that as an input to the Model for inference. Result: Getting detections with some confidences.
```python
model = OwlViTForObjectDetection.from_pretrained(...)
processor = OwlViTProcessor.from_pretrained(...)

inputs = processor(text=texts, images=image, return_tensors="pt")
outputs = model(**inputs)
```

**Additional flow**
It would be useful to have additional capability to support images embeddings as an input (either for a processor or the model itself), so the additional flow would look like this:
take 100 (e.g.) images -> run to calculate images embeddings, save (a dataframe, or files) -> use model(embeddings_dir, text_terms) to infer detections.
```python
model = OwlViTForObjectDetection.from_pretrained(...)
processor = OwlViTProcessor.from_pretrained(...)

# using images_embeddings instead of images
inputs = processor(text=texts, images_embeddings=image_embedding, return_tensors="pt")
outputs = model(**inputs)
```

### Motivation

Creating a cache, precomputed set of images embeddings for faster inference / search by text.
It is possible, that this functionality already exists, but because of the way owl-vit is structured, it might be tricky to perform: currently **inputs contain embeddings for bboxes and text.

### Your contribution

Would be difficult, as I'm not familiar with the infra.

Comments:
cc @alaradirik and @NielsRogge 

Hi @ramanova, thanks for the suggestion! 

This is doable but might be a bit tricky. `OwlViTProcessor` preprocesses images (resizing, cropping, etc.) and doesn't compute embeddings so you would need compute the embeddings using the base `OwlViTModel`. 

@NielsRogge do you know if there are any other models that provide this kind of functionality?

Several NLP models, like BERT, provides the `inputs_embeds` argument as seen [here](https://github.com/huggingface/transformers/blob/d066c3731bed1755f93ea64f0f00981b805532de/src/transformers/models/bert/modeling_bert.py#L919), which allows you to provide embeddings yourself rather than `input_ids`. So the use case here is similar, I assume.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Set task and language tokens for whisper model
Author: bofenghuang
Pull request: no

### Feature request

Hi @ArthurZucker, thanks for the great work on whisper models.

I would like to know if it's possible to also have a `set_prefix_tokens` function in `WhisperForConditionalGeneration`, which receives the language/task name and changes the language/task token in `model.config.forced_decoder_ids`, in order to run the ASR inference on languages other than EN. 

As far as I know it has by default `['
', '
', '
']`, so I have to get the language token ID first and set it manually before running generate



### Motivation

An easy API to set language/task would be useful.

### Your contribution

Willing to do if this function doesn't exist

Comments:
Hey! That's a good idea and also on my TODO! 
It is debatable whether we should have this in the `WhisperForConditionalGeneration` or add a new class for multilingual Decoding which would do this automatically. We could also add a `WhisperForSequenceClassification` class which would just detect the language. 

CC @sgugger and @patrickvonplaten as this is really a design question. 

I think this is something we would want to solve with generate use-case specific configurations (cc @gante) 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Models cannot be loaded if they have dot "." in name
Author: j-adamczyk
Pull request: no

### System Info

- `transformers` version: 4.24.0
- Platform: Linux-5.14.0-1054-oem-x86_64-with-glibc2.31
- Python version: 3.9.15
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.13.0+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@NielsRogge @LysandreJik

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Try to use `google/mobilenet_v2_1.0_224`:
```
    model = AutoModelForImageClassification.from_pretrained(
        "google/mobilenet_v2_1.0_224",
        num_labels=2,
        ignore_mismatched_sizes=True,
    )
```

I get:
```
Traceback (most recent call last):
  File "train.py", line 217, in <module>
model = AutoModelForImageClassification.from_pretrained(
  File "/opt/conda/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py", line 434, in from_pretrained
config, kwargs = AutoConfig.from_pretrained(
  File "/opt/conda/lib/python3.8/site-packages/transformers/models/auto/configuration_auto.py", line 796, in from_pretrained
config_class = CONFIG_MAPPING[config_dict["model_type"]]
  File "/opt/conda/lib/python3.8/site-packages/transformers/models/auto/configuration_auto.py", line 503, in __getitem__
raise KeyError(key)
KeyError
: 'mobilenet_v2'
```

I suspect that this happens due to line 790 in `configuration_auto.py`:
```
module_file, class_name = class_ref.split(".")
```

Since this model has a dot "." in name, it is splitted here and library is looing for `mobilenet_v2` instead, which does not exist.

### Expected behavior

Load a model from HuggingFace Hub.

Comments:
Hi, @j-adamczyk. The problem here is that `transformers` version that includes `MobileNetV2` model has not yet been released to [PyPi](https://pypi.org/project/transformers/#history) etc., hence there's no `'mobilenet_v2'` key. 
Though you can install the `transformers` from source, and start using `MobileNetV2` right after.

To install from [source](https://huggingface.co/transformers/v3.5.1/installation.html#installing-from-source), clone the repository and install with the following commands:
```
git clone https://github.com/huggingface/transformers.git
cd transformers
pip install -e .
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Closing as this issue seems resolved.
my colab do not load new notebook it has bug
Author: shahrzad-setayesh
Pull request: no

### System Info

#hi
i am university student and need colab please help me i can not open new notebook in colab .

do not load  new notebook ,....
thanks
shahrzad.setayesh88@gmail.com

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

hi
i am university student and need colab please help me i can not open new notebook in colab .

### Expected behavior



do not load  new notebook ,....
thanks
shahrzad.setayesh88@gmail.com

Comments:
Use `baddbmm` to reduce the number of kernel calls when running T5
Author: thomasw21
Pull request: yes

# What does this PR do?

Reduce the number of kernel calls by using `baddbmm` and built-in `F.softmax(..., dtype=torch.float))`

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Arf turns out this doesn't work without having extra copies. Closing as I'm not sure if it's worth it.
GPT Neo - no attention weights scaling in pytorch implementation of GPT Neo
Author: FuTSy13
Pull request: no

It seems that there is no scaling of attention weights in GPT-Neo implementation of SelfAttention. Probably https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt_neo/modeling_gpt_neo.py#L188  should be modified as follows:
```
attn_weights = torch.matmul(query, key.transpose(-1, -2))
scale = self.head_dim ** -0.5
attn_weights = attn_weights * scale
```


Comments:
Do you have a source available that says they do use scaling here? Though it is common practice, I don't think it is necessarily required.

Yes, it seems you are right. Originally they use mesh_tensorflow realization of multi-head self-attention in which i don't see such scaling.

But at the same time in flax realization (https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt_neo/modeling_flax_gpt_neo.py#L230) flax.linen.attention.dot_product_attention_weights is used. And in its source code there is scaling. But i don't think that this incosistency is crucial.
fix typo output not ouput in bitsandbytes trainer test
Author: Thomas-MMJ
Pull request: yes


# What does this PR do?
fixes a typo in the trainer test for bitsandbytes that was causing an error on pytest collection



## Before submitting
- [ yes] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add missing information on token_type_ids for roberta model
Author: raghavanone
Pull request: yes

# What does this PR do?

Add missing information on token_type_ids for roberta model

Fixes #19744

- [X ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?

## Who can review?

@sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Thanks for the PR. Can you be a bit more explicit as the configuration will always be initialized with this parameter. Maybe it needs to be 2 for instance? Or any vlaue >= 2?

Done
Change the import of kenlm from github to pypi
Author: raghavanone
Pull request: yes

# What does this PR do?

Fixes #19686 

## Who can review?

@patrickvonplaten 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Thanks for your PR. The main interest of having the package on PiPy is that we can put it in our setup and not have to write a specific command to install it. You should add it to the extras for audio.

Done

Please remove all manual install of `kenlm`. I don't know if I was unclear in my previous comments, but putting it in the setup will make it installed when we do `pip install -e .[xxx]`, so there is no need for a separate install line.

> Please remove all manual install of `kenlm`. I don't know if I was unclear in my previous comments, but putting it in the setup will make it installed when we do `pip install -e .[xxx]`, so there is no need for a separate install line.

Oh, I misunderstood who you put in earlier comment. now removed.

I think we still have 2 files to be clean up ?

- docker/transformers-pytorch-gpu/Dockerfile
- docker/transformers-doc-builder/Dockerfile
change the way sentinel tokens can retrived
Author: raghavanone
Pull request: yes

# What does this PR do?

fixes the issue #19298 

@SaulLu 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@SaulLu  @sgugger  Done

As pointed above, you're still missing the stronger test to detect the sentinel tokens.

Thanks a lot for working on this. Maybe I'm just nitpicking, but wouldn't it be great if the returned sentinel tokens were sorted?

```
self.tokenizer.get_sentinel_tokens()
['<extra_id_1>', '<extra_id_26>', '<extra_id_55>', '<extra_id_87>', '<extra_id_50>', '<extra_id_49>', '<extra_id_74>', '<extra_id_66>', '<extra_id_83>', '<extra_id_30>', '<extra_id_3>', '<extra_id_0>', '<extra_id_90>', '<extra_id_14>', '<extra_id_71>', '<extra_id_6>', '<extra_id_18>', '<extra_id_4>', '<extra_id_75>', '<extra_id_99>', '<extra_id_63>', '<extra_id_58>', '<extra_id_48>', '<extra_id_62>', '<extra_id_73>', '<extra_id_20>', '<extra_id_70>', '<extra_id_21>', '<extra_id_38>', '<extra_id_34>', '<extra_id_88>', '<extra_id_28>', '<extra_id_97>', '<extra_id_91>', '<extra_id_65>', '<extra_id_81>', '<extra_id_98>', '<extra_id_23>', '<extra_id_96>', '<extra_id_12>', '<extra_id_19>', '<extra_id_79>', '<extra_id_78>', '<extra_id_68>', '<extra_id_95>', '<extra_id_35>', '<extra_id_42>', '<extra_id_27>', '<extra_id_85>', '<extra_id_67>', '<extra_id_17>', '<extra_id_36>', '<extra_id_93>', '<extra_id_37>', '<extra_id_60>', '<extra_id_77>', '<extra_id_32>', '<extra_id_92>', '<extra_id_33>', '<extra_id_40>', '<extra_id_86>', '<extra_id_53>', '<extra_id_10>', '<extra_id_31>', '<extra_id_72>', '<extra_id_24>', '<extra_id_80>', '<extra_id_13>', '<extra_id_45>', '<extra_id_61>', '<extra_id_52>', '<extra_id_8>', '<extra_id_44>', '<extra_id_57>', '<extra_id_16>', '<extra_id_84>', '<extra_id_51>', '<extra_id_56>', '<extra_id_41>', '<extra_id_64>', '<extra_id_39>', '<extra_id_9>', '<extra_id_25>', '<extra_id_59>', '<extra_id_46>', '<extra_id_7>', '<extra_id_54>', '<extra_id_29>', '<extra_id_89>', '<extra_id_22>', '<extra_id_15>', '<extra_id_43>', '<extra_id_47>', '<extra_id_76>', '<extra_id_5>', '<extra_id_94>', '<extra_id_11>', '<extra_id_82>', '<extra_id_2>', '<extra_id_69>']
```
batch_encode_plus() causes OOM, while encode_plus does not 
Author: yuhongqian
Pull request: no
Labels: name: wontfix

name: Core: Tokenization

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I am running a sequence classification task using `DistilBertForSequenceClassfication`. I follow `examples/text_classfication/run_glue.py` and `src/transformers/data/processors/glue.py` to implement my data loading process. My dataset is a rather large one (~2.5 GB with 7M+ examples), compared to those of the GLUE tasks. 

In the current `glue.py`, `_glue_convert_examples_to_features()` reads all the examples into a list, and then call `batch_encode_plus()` on that list. On my large dataset, this implementation caused an out-of-memory (OOM) error. Therefore, I switched to `encode_plus()`, and called it on individual data example while looping through the dataset. `encode_plus()` did not cause OOM. 

I wonder if there is something wrong with `batch_encode_plus()` so that it cannot handle all the examples in a dataset at once? If that is the case, it might be a good idea to add a corresponding note to the documentation. 



Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I'm also running out of memory using `BertTokenizerFast.batch_encode_plus()`. I'm using the `BertTokenizer.batch_encode_plus()` now and it seems to be working but its very slow and single-threaded! I have 220 GB RAM and the dataset is under 2 GB 😞 .

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Any new developments?

any solutions to this? facing the same issue
TF AdamWeightDecay fix for 2.11
Author: Rocketknight1
Pull request: yes

The TF changelog said that the optimizer had been moved to `tf.keras.optimizer.legacy`, but the true path is `tf.keras.optimizers.legacy`. Because of the conditional in the PR, we didn't notice the error, but it's resolved now!

Fixes #20847 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Position_ids misuses self.padding_idx
Author: Rocketknight1
Pull request: no
Labels: name: bug

### System Info

issue is platform-independent

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://colab.research.google.com/drive/1ZuL1pc0fMBOnJahF18xArcFihfAmLwDo?usp=sharing

### Expected behavior

I'm self-assigning this and will file a PR - I'm just creating this issue so there's a record of it and I don't lose track! The issue is that the `position_ids` in several models (`RoBERTa` and any models that copied code from it) misuse `self.padding_idx` in at least two ways. Firstly, `self.padding_idx` is passed as the padding_idx to self.position_ids. This means that the position embedding in that position is never updated, even though there is no such thing as a padding token in position embeddings.

Secondly, `create_position_ids_from_inputs_embeds`, which was also written for `RoBERTa` before being copied to several other models initializes the position IDs starting from `self.padding_idx`, which does not make sense. This can cause a crash (see Colab above) when the model's padding_idx is not a low integer value, but even when it doesn't crash the model it creates incorrect position IDs whenever the user passes input embeds instead of input IDs.

Comments:
Update: I think this is actually reproducing behaviour in the original RoBERTa, and isn't a bug after all. It's probably unnecessary when copied to all the other models out there, but it's not the end of the world as long as everyone keeps their padding_idx low.

@Rocketknight1 I ran through the same issue while training `XLMRobertaForMaskedLM`.
Here's the issue details:

1. The default padding token id is `1`
2. `create_position_ids_from_input_ids` will set position id of padding tokens to `1`, and the actual sequence starts at `2`
3. For input sequences of length truncated to max length, (defaults to `512`), the position id for the last token will be `513`
4. While creating positional embeddings, token with position id `513` (should be `512`) will through an indexing exception

I believe this needs to be addressed either by:

- XLMPreTrainedTokenizer and set padding token's id to `0` instead of `1`
- `create_position_ids_from_input_ids` implementation


@YazanShannak Thank you for the report - this is similar to what I observed as well. Unfortunately, I don't think we can change the padding token ID for RoBERTa and RoBERTa-based models without breaking compatibility with the original implementation.

My suspicion is that the code for `create_position_ids_from_input_ids` is actually correct for RoBERTa, but was copied to other models and causes issues there. Can you check if you get the same error when using RoBERTa instead of XLMRoBERTa?
Clarify `use_fast` parameter in docstring
Author: stevhliu
Pull request: yes

This PR addresses the ambiguity of the `use_fast` parameter raised in #20817.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> what happens if the architecture supports it but the model doesn't?

Hmm that's a good question! Do you happen to know if any of the other architectures have this issue or if it is just a bug with OPT? I'll remove the suggestion to check the supported framework list so we don't end up confusing anyone.

It's the first time I run into this inconsistency across models of the same arch, but I have never needed to ensure it was `fast` before so who knows it may have happened a lot and I wasn't the wiser.

So +1 to remove the suggestion unless we somehow can stand behind it.

This was the bug for OPT, so it shouldn't happen for other models.
Embed circle packing chart for model summary
Author: stevhliu
Pull request: yes

This PR embeds an interactive chart of the most popular models by modality so users have a nice high-level visual overview of the 🤗 Transformers modelscape.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I think the number of downloads dictates the size, right?

Impressive! I like how visual it is. Makes it understandable straight away. I guess a potential improvement would be to be able to read the subcategories without first clicking on a major category, but I don't know how feasible that is. Thanks for working on this! Really cool.

Thanks for the feedback!

I think it might make the visual more difficult to read if we also included the subcategories (modality) in addition to the main category. For example, the decoders bubble is already quite small, and adding more text might make it more cluttered (same for some of the smaller encoder-decoder bubbles).