noamaanMulla-03/transformers-issues huggingface dataset:
`accelerate` support for `OwlViT`
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR adds `accelerate` support for `OwlViT` so that any model from this family can be loaded in `8-bit`! Here is a small snippet on how to load and run the model in `8-bit` (based on the snippet from the model card):
```
# pip install accelerate bitsandbytes
import requests
from PIL import Image
import torch

from transformers import OwlViTProcessor, OwlViTForObjectDetection

processor = OwlViTProcessor.from_pretrained("google/owlvit-base-patch32")
model = OwlViTForObjectDetection.from_pretrained("google/owlvit-base-patch32", device_map="auto", load_in_8bit=True)

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
texts = [["a photo of a cat", "a photo of a dog"]]
inputs = processor(text=texts, images=image, return_tensors="pt")
outputs = model(**inputs)

# Target image sizes (height, width) to rescale box predictions [batch_size, 2]
target_sizes = torch.Tensor([image.size[::-1]])
# Convert outputs (bounding boxes and class logits) to COCO API
results = processor.post_process(outputs=outputs, target_sizes=target_sizes)

i = 0  # Retrieve predictions for the first image for the corresponding text queries
text = texts[i]
boxes, scores, labels = results[i]["boxes"], results[i]["scores"], results[i]["labels"]

# Print detected objects and rescaled box coordinates
score_threshold = 0.1
for box, score, label in zip(boxes, scores, labels):
    box = [round(i, 2) for i in box.tolist()]
    if score >= score_threshold:
        print(f"Detected {text[label]} with confidence {round(score.item(), 3)} at location {box}")
>>> Detected a photo of a cat with confidence 0.705 at location [321.25, 19.8, 643.12, 376.88]
>>> Detected a photo of a cat with confidence 0.729 at location [0.94, 53.55, 319.69, 473.91]
```
Added also a slow test to make sure users can run inference in `fp16` (`8bit` converts the model in. `fp16` under the hood)

All slow tests pass except the one mentioned in #20410 that should be fixed once merged

cc @alaradirik @sgugger @NielsRogge 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks everyone!
trying fine-tuning Reformer on Squad2 dataset from pre-trained model "google/crime-and-punishment".
Author: FrancescoTroiano
Pull request: no

I'm trying fine-tuning Reformer on Squad2 dataset from pre-trained model "google/crime-and-punishment". 
Using tok.cls_token = tok.pad_token, I have the following error:  
![image](https://user-images.githubusercontent.com/75449189/196391419-7387c255-cc07-4f81-975d-da544fef95c4.png)
So I add tok.pad_token = tok.eos_token, but I have a new error: 2 is not in list.
can someone help me? Thank you

_Originally posted by @FrancescoTroiano in https://github.com/huggingface/transformers/issues/5436#issuecomment-1282092684_
      

Comments:
Please follow the template to file your issue or ask your question on the [forums](https://discuss.huggingface.co/).

Ok, thank you. I open a discussion in the forum in the Transformers category about that

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Implementing SHAP algorithm on visualBERT transformer 
Author: MUZAMMILPERVAIZ
Pull request: no

### System Info

Hi @LysandreJik , @NielsRogge, @sgugger,
I am working to apply [shap](https://shap.readthedocs.io/en/latest/index.html) algorithm on visualbert. I found a piece of code that run`s well on distilbart-xsum-12-6 , Here is the code:

```
import numpy as np
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import shap
import torch
# load transformer language model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("sshleifer/distilbart-xsum-12-6")
model =  AutoModelForSeq2SeqLM.from_pretrained("sshleifer/distilbart-xsum-12-6").cuda()
s=["In this picture, there are four persons: my father, my mother, my brother and my sister."]

explainer = shap.Explainer(model,tokenizer)

shap_values = explainer(s)
![image](https://user-images.githubusercontent.com/69303067/197922659-c1f3cee3-94cb-429d-8123-0920ef0477fd.png)
```

But i don`t know how to implement the same thing on visualBERT. Is there any repository which demonstrate the implementation of shap algorithm on visualBERT transformer or anyone know how to do this? 
Thanks for your time. 

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import BertTokenizer, VisualBertForPreTraining, VisualBertForQuestionAnswering
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
#model = VisualBertForPreTraining.from_pretrained('uclanlp/visualbert-nlvr2-coco-pre')
model = VisualBertForQuestionAnswering.from_pretrained("uclanlp/visualbert-vqa")
from datasets import load_dataset
dataset = load_dataset("textvqa")
explainer = shap.Explainer(model,tokenizer)
shap_values = explainer(dataset['train'][0]['question'])
````

### Expected behavior

![image](https://user-images.githubusercontent.com/69303067/197923767-bcf32626-1970-40ca-8b67-b74735c9d8f1.png)


Comments:
You should use the [forums](https://github.com/huggingface/safetensors/pull/34) for questions like this as we keep issues for bugs and feature requests only.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Closing this issue for the reason above.
[trainer] apex test fix
Author: stas00
Pull request: yes

there was a small mistake in a test of https://github.com/huggingface/transformers/pull/18961, this PR fixes it.

the main CI doesn't have apex installed that's why it missed it.

Thank you, @ydshieh for the heads up about the breakage


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Could you do `pip show huggingface_hub`?
Author: wccccp
Pull request: no

        Could you do `pip show huggingface_hub`?

_Originally posted by @NielsRogge in https://github.com/huggingface/transformers/issues/20447#issuecomment-1327215502_
      

Comments:
This issue appears to have been opened in error. If this is the case @wccccp you can close this Issue Request using the buttons below 👇 

> This issue appears to have been opened in error. If this is the case @wccccp you can close this Issue Request using the buttons below 👇

thank you
Add CLIPSeg
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds CLIPSeg, a nice extension of CLIP for zero-shot and one-shot (image-guided) image segmentation.

To do:

- [x] transfer checkpoints and update code
- [x] update base_model_prefix

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Is the CLIPSeg yet to be released in the latest version?
How to make gpt2 accurately tokenize lowercase entities?
Author: ahxchzt
Pull request: no

Many datasets pre-lowercase their text content, so some entities are lowercase in these datesets, like:

**"paris"**

For GPT2Tokenizer, it can tokenize "Paris" to "Paris" but it tokenizes **"paris" to "par" and "is".** 
How to fix it? Is there a uncased GPT2?
Thanks!

Comments:
Please use the [forums](https://discuss.huggingface.co/) for such questions, as we keep the issues for bugs and feature requests only.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
add greek translation to index
Author: ArchontisKostis
Pull request: yes

# What does this PR do?

Fixes #19788

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19789). All of your documentation changes will be reflected on that endpoint.

Yes, Greek under ISO 639-1 would be `el`, which is what we use. `gre` corresponds to 639-2. 

(source: https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes)

Hello! I am really sorry for the mistake I will fix it shortly!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Sorry missed the change! It looks like there is a problem with CircleCI (tests are not run). Could you try refreshing your permissions as shown [here](https://support.circleci.com/hc/en-us/articles/360048210711-How-to-Refresh-User-Permissions-)?

Could you also add the Greek language code [here](https://github.com/huggingface/transformers/blob/11f3ec7224c83c9e5c379a774b9d3984e68d26fa/.github/workflows/build_documentation.yml#L18) and [there](https://github.com/huggingface/transformers/blob/11f3ec7224c83c9e5c379a774b9d3984e68d26fa/.github/workflows/build_pr_documentation.yml#L17) so that the doc is built in Greek?

let us know if you need any help, thanks!
[fix bug] Although this bug will not have any impact on rag here (bec…
Author: anonNo2
Pull request: yes

…ause em is used as the evaluation metric), if you add bleu or rouge here, this bug will have an incorrect impact

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)   
There is a small bug in the rag implementation, the two parameters are reversed.(pred and hypo)
This error will not affect the experimental results here, but if you increase the evaluation metrics such as bleu or rouge, it will affect the results.


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Error while importing pretrained model
Author: spulick
Pull request: no

### System Info

- `transformers` version: 4.23.1
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.10.6
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.13.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@patil-suraj @patrickvonplaten @LysandreJik 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained("gpt2", num_labels=2)
```
The above code, on execution, returns
```
RuntimeError: Failed to import transformers.models.gpt2.modeling_gpt2 because of the following error (look up to see its traceback):
name '_C' is not defined
```

This error is not specific to that model or to two labels. I ran the example code snippet given at https://huggingface.co/docs/transformers/training#train-with-pytorch-trainer. The same error is presented, albeit with a different model name.

### Expected behavior

From the tutorial, I was expecting a pretrained model object to be initialized, so that I could proceed with fine-tuning it.

Comments:
Hi @orectique. Have you tried to do something like the following:
https://discuss.pytorch.org/t/nameerror-name-c-is-not-defined-while-importing-torch/124721/2 
or
https://github.com/pytorch/pytorch/issues/1633#issuecomment-323435572

Thank you, @atturaioe. I explored both of those forums and tried the strategies. However, the error seems to have been something else. Anywho, in exasperation, I dumped the entire environment I was using and installed all the packages from scratch. That seems to have done it.
Why this dummpy input is not consistent with pytorch
Author: MichaleDong
Pull request: no

https://github.com/huggingface/transformers/blob/bd469c40659ce76c81f69c7726759d249b4aef49/src/transformers/models/layoutlmv3/modeling_tf_layoutlmv3.py#L988

The dummpy input of pytorch layoutlmv3 is like this, from transformers.utils:
https://github.com/huggingface/transformers/blob/v4.23.1/src/transformers/modeling_utils.py#L954 

from transformers.utils import DUMMY_INPUTS
>>> print(DUMMY_INPUTS)
[[7, 6, 0, 0, 1], [1, 2, 3, 0, 0], [0, 0, 0, 4, 5]]




Comments:
Hi there! Please ask questions like this on the [forums](https://discuss.huggingface.co/), as we keep issues for bugs and feature requests only.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix doctests for audio models
Author: ydshieh
Pull request: yes

# What does this PR do?

The condition was wrong (`[]` should be `()`) and failed some doctests as they get wrong classes

```python
if ["SequenceClassification" in model_class or "AudioClassification" in model_class]
```
should be
```python
if ("SequenceClassification" in model_class or "AudioClassification" in model_class)
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Merge as the failing (TF) test is irrelevant to this PR.
Add missing tokenizer tests - RemBert
Author: IMvision12
Pull request: yes

# What does this PR do?
Fixes : #16627
Added tokenizer tests for Rembert. I took reference from test_tokenization_camembert.py

@SaulLu @LysandreJik

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@IMvision12 It looks like the added tests do not pass. You can try locally with
```
pytest tests/models/rembert/test_tokenization_rembert.py
```

While using sentencepiece.model :

`SAMPLE_VOCAB = get_tests_dir("fixtures/test_sentencepiece.model")`
these tests are failing : 

- FAILED tests/models/rembert/test_tokenization_rembert.py::RemBertTokenizationTest::test_convert_token_and_id - AssertionError: '<unk>' != '[PAD]'
- FAILED tests/models/rembert/test_tokenization_rembert.py::RemBertTokenizationTest::test_get_vocab - AssertionError: '<unk>' != '[PAD]'
[ROC_BERT] Make CI happy
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes a small slow test that was failing when trying to play with 8-bit conversion for BERT models

cc @ydshieh 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20175). All of your documentation changes will be reflected on that endpoint.

Hi @younesbelkada Thank you for trying make CI happy.

But I am not sure why we need `torch.allclose` for 2 integer tensors. This should be used for float tensors. For the integer outputs (here token ids), they should be equal, so `self.assertEqual` would be the one to use I think.

Thanks @ydshieh for the heads up, indeed I believe we can use that. Let me update it

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20175). All of your documentation changes will be reflected on that endpoint.

ahah merci beaucoup @ydshieh !!
Safetensors offload
Author: sgugger
Pull request: yes

# What does this PR do?

This PRs make offload to disk more efficient for models that have a checkpoint in safetensors format: instead of re-saving everything as Numpy memory-mapped array, it uses directly the fact we can access the tensor in the checkpoint without loading the rest.

Goes with https://github.com/huggingface/accelerate/pull/873

Comments:
_The documentation is not available anymore as the PR was closed or merged._
fix both failing RoCBert tests
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fixes two failing test : 
- tokenization test did not take into account that the dummy tokenizer's `pad_token_id = 2` and `bos_token_id = 1`. The values used were `102, 101`. 
- on of the modeling test used `assertEqual` on tensors and not list, which is ambigus. 
 

Comments:
The two tests no pass locally (which was not the case before) 

_The documentation is not available anymore as the PR was closed or merged._
[AutoBackbone] Improve API
Author: NielsRogge
Pull request: yes

# What does this PR do?

As backbones themselves have hidden states and optional attentions, this PR adds them to the `BackboneOutput`.

This way, frameworks that leverage backbones can return hidden states/attentions of the backbone if the user specifies `output_hidden_states=True` or `output_attentions=True`.

To do:

- [x] perhaps we should test backbones with all common tests

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Pinging @ydshieh on this PR as I made 2 updates to test_modeling_common.py to support models which output a tuple of tensors as their first output. I've updated `test_determinism` and `test_save_load` to make them more general.

@michaelbenayoun as seen on the CI, the torch fx tests fail because backbones aren't supported yet.

Could you add support for them in a separate PR?

> Pinging @ydshieh on this PR as I made 2 updates to test_modeling_common.py to support models which output a tuple of tensors as their first output. I've updated `test_determinism` and `test_save_load` to make them more general.

Looks good for me regarding the tests.

I would personally put the check `isinstance(first, tuple)` inside the `check_xxx` methods, and call them recursively.
This way we don't need to worry if each element in the list/tuple would contain list/tuple. But no obligation for now.

@ydshieh for some reason the CI isn't run when I push new commits, do you know why?

Not really, try again with a new empty commit?

git commit --allow-empty -m "empty commit to trigger CI"


@ydshieh Thanks a lot! @sgugger feel free to approve :)
Include image processor in add-new-model-like
Author: amyeroberts
Pull request: yes

# What does this PR do?

* Adds logic to `add-new-model-like` CLI to include image processors
* Updates `tests/utils/add_new_model_like.py` so tests run (was outdated) 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@ydshieh The test requires all three frameworks (PyTorch, TensorFlow and Flax) so it is never run in the CI.

OK, I completely misunderstand `Add model like runner / Add new model like template tests (pull_request)`.
Ignore my previous comment 🙏 .
chore: add link to the video cls notebook.
Author: sayakpaul
Pull request: yes

We recently added a [notebook](https://github.com/huggingface/notebooks/blob/main/examples/video_classification.ipynb) that shows how to fine-tune the VideoMAE model on a custom dataset. This PR adds the notebook link to the model doc. 

Cc: @osanseviero 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@NielsRogge I don't have the rights to merge the PR. If you have, could you perform the duties?