noamaanMulla-03/transformers-issues huggingface dataset:
Add type hints for Luke in PyTorch
Author: bhavika
Pull request: yes

# What does this PR do?

Adds type hints for Luke https://github.com/huggingface/transformers/issues/16059 

Tagging @Rocketknight1 for review! 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Improve model variable naming - CLIP [TF] 
Author: bhavika
Pull request: yes

# What does this PR do?


Uses the @unpack_inputs decorator instead of `input_processing` for CLIP (TF). See https://github.com/huggingface/transformers/issues/16051

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Tagging @Rocketknight1 for review! 👋🏽 



Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc @patil-suraj 

Also, making a cheeky edit on your PR description -- I'm swapping `Fixes` with `See` (`Fixes` closes the associated issue)

Thanks for the review @gante! Made changes in https://github.com/huggingface/transformers/pull/16128/commits/2840ef018073dd7df88a2e85e193dfbca775b88b and I'm confirming that `RUN_SLOW=1 py.test -vv tests/[model_name]/test_modeling_tf_[model_name].py` works for `clip`.
Dcoker images runtime -> devel
Author: LysandreJik
Pull request: yes

Updates the base image for the docker images used for testing to use the `devel` version of the original docker images instead of the `runtime` version, as it's the version necessary for some TF utilities to run.

Comments:
Let me just add a failing example before this PR:
(just for the record)

```
tests/bart/test_modeling_tf_bart.py::TFBartModelTest::test_xla_mode
(line 54)  tensorflow.python.framework.errors_impl.InternalError: libdevice not found at ./libdevice.10.bc [Op:__inference_run_in_graph_mode_1125293]
```
Resnet
Author: FrancescoSaverioZuppichini
Pull request: yes

# What does this PR do?
This WIP PR adds [ResNet](https://arxiv.org/abs/1512.03385). 

Currently, the model can be used as follows

```python
import requests
from io import BytesIO
res = requests.get('https://github.com/huggingface/transformers/blob/master/tests/fixtures/tests_samples/COCO/000000039769.png?raw=true')
image = Image.open(BytesIO(res.content))

feature_extractor = AutoFeatureExtractor.from_pretrained("Francesco/resnet50")
model = ResNetForImageClassification.from_pretrained("Francesco/resnet50").eval()

inputs = feature_extractor(image, return_tensors="pt")
outputs = model(**inputs)
print(model.config.id2label[torch.argmax(outputs.logits).item()])
# tiger cat
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15770). All of your documentation changes will be reflected on that endpoint.

I've removed the `regression` loss from `ForImageClassification` since it is not a loss used for image classification. This open for discussion

I've removed the `D` variant code, it will be added by a different PR. Three new models outputs have been added: `BaseModelOutputWithNoAttention`, `BaseModelOutputWithNoAttentionAndWithPooling` and `ImageClassificationModelOutput`

Address most of the comments, the blocking ones are related to outputs docstring
Add type annotations for Beit, ViT and Deit models - Pytorch
Author: johnnv1
Pull request: yes

# What does this PR do?

I added type annotations for  Beit, ViT and Deit models at PyTorch as described in #16059 

@Rocketknight1 

At Beit model, have some missing annotations because the annotation is another class at the file that are declared after. Also, at `BeitSelfOutput` class, I don't know the correct type for the gamma parameter at `forward` method -- the `gamma`  and `input_tensor` seems not to be used, but I prefer not to modify this now.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi, thank you for this! I won't merge it yet because I'm going to ask the other team members on Monday what we think about the missing annotations - it's possible to use [forward references](https://peps.python.org/pep-0484/#forward-references) for this, but I want to make sure that plays well with our code analysis tools before we do that.

Already have a command that runs something like `mypy` to check the types?

Hi @johnnv1, I just confirmed with the team that forward references are fine for types that aren't defined yet!

@Rocketknight1 after rebase upstream/master the `make fixup` shows that `vit_mae` need to be updated because of the `# Copied from ...`, but when I run `make fix-copies` he has been adding the `ViTConfig` on the config type annotation, I think these need to be `ViTMAEConfig`, what is the right way to deal with this? 

Hi @johnnv1, that rebase made this pretty tricky to review! It might be easier to close this and start a new PR and just copy the changes over? If there are classes that are copies then the original source of the copy should be changed, followed by `make fix-copies`. If other people are working on the classes you need to change, it might be easiest to wait until those PRs are done before starting yours!
Add type hints for GPTNeo PyTorch
Author: Tegzes
Pull request: yes

Adding type hints for forward methods in user-facing class for GPTNeo (PyTorch) as mentioned in #16059  

@Rocketknight1 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

If you run `make style` from the root of the repo and push again then it should fix the failing the tests.

@patil-suraj Thanks for handling this one, feel free to merge it whenever you're happy!

@Tegzes if you change the type for `past_key_values` from `List` to `Tuple` we should be good to merge :) 

Awesome, I'm making the changes right now
RuntimeError: [enforce fail at CPUAllocator.cpp:68] Out of memory during batched inference
Author: kaushalshetty
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.16.2
- Platform: p3.24xlarge
- Python version: 3.6
- PyTorch version (GPU?): 1.10.0+cu102
- Tensorflow version (GPU?): 2.6
- Using GPU in script?: No 
- Using distributed or parallel set-up in script?: No


 @LysandreJik 



## Information

I am using codebert model(RoBerta) for inference by wrapping it using huggingface. Below, I am trying to retrieve and store the  embeddings that is output from the model.
Code:

```
from transformers import RobertaTokenizer, RobertaModel
tokenizer = RobertaTokenizer.from_pretrained("microsoft/codebert-base")
# this is the pretrained model which we plan to fine-tune/transfer-learn further
model = RobertaModel.from_pretrained("codebert")
batch_size = 8
# Data is already loaded
tokens = tokenizer(data, return_tensors="pt", truncation=True, max_length=768, padding=True)["input_ids"]
conc = torch.empty(size=(len(tokens), 768))
model.eval()
for i in range(0, len(tokens), batch_size): 
    conc[i:i+batch_size] = model(tokens[i:i+batch_size])[1]
```
The shape of tokens is 5000 * 768 . 
System is a p3.24xlarge machine with about 700GB of RAM. 
Why is the model giving "DefaultCPUAllocator: can't allocate memory" even after batching the input and the system having large RAM?


Comments:
Have you tried with a smaller amount of tokens to see where it fails?

Yes, I ran the "top" command in shell and observed that memory almost maxes out to 99% for just 2000. But, I was able to get over the memory issue by detaching the tensor and converting it to cpu() and then numpy() . 
```model(tokens[i:i+batch_size])[1].detach().cpu().numpy()```. My problem is fixed but just wondering what was eating up the memory.
Visual Attention Network (VAN)
Author: FrancescoSaverioZuppichini
Pull request: yes

# What does this PR do?
This PR adds [Visual Attention Network](https://arxiv.org/pdf/2202.09741.pdf). 

Currently, the model can be used as follows

```python
import requests
from io import BytesIO
res = requests.get('https://github.com/huggingface/transformers/blob/master/tests/fixtures/tests_samples/COCO/000000039769.png?raw=true')
image = Image.open(BytesIO(res.content))

feature_extractor = AutoFeatureExtractor.from_pretrained("zuppif/van-base")
model = VanForImageClassification.from_pretrained("zuppif/van-base").eval()

inputs = feature_extractor(image, return_tensors="pt")
outputs = model(**inputs)
print(model.config.id2label[torch.argmax(outputs.logits).item()])
# tabby, tabby cat
```

## TODO
- [x] modeling
- [x] weights
- [x] doc
- [x] tests


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for the reviews. I've resolved all the comments that can be resolved and asked the authors if we can create an organization for them in the hub.
Improve Swin for VisionEncoderDecoder
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR improves support for the Swin Transformer with the `VisionEncoderDecoderModel` framework.

Thanks to @FrancescoSaverioZuppichini we now have general formulas to compute the sequence length and hidden size of the last hidden state of Swin Transformer.



Comments:
_The documentation is not available anymore as the PR was closed or merged._
Make transformers.utils.fx. _SUPPORTED_MODELS unique
Author: pbelevich
Pull request: yes

Currently this tuple contains duplicates(e.g. `RobertaForMaskedLM` or `GPT2LMHeadModel`): `AlbertModel, AlbertForPreTraining, AlbertForMaskedLM, AlbertForMultipleChoice, AlbertForQuestionAnswering, AlbertForSequenceClassification, AlbertForTokenClassification, BertModel, BertForPreTraining, BertForNextSentencePrediction, BertForMaskedLM, BertLMHeadModel, BertForMultipleChoice, BertForQuestionAnswering, BertForSequenceClassification, BertForTokenClassification, DistilBertModel, DistilBertForMaskedLM, DistilBertForMaskedLM, DistilBertForMultipleChoice, DistilBertForQuestionAnswering, DistilBertForSequenceClassification, DistilBertForTokenClassification, MobileBertModel, MobileBertForPreTraining, MobileBertForNextSentencePrediction, MobileBertForMaskedLM, MobileBertForMultipleChoice, MobileBertForQuestionAnswering, MobileBertForSequenceClassification, MobileBertForTokenClassification, ElectraModel, ElectraForPreTraining, ElectraForMaskedLM, ElectraForCausalLM, ElectraForMultipleChoice, ElectraForQuestionAnswering, ElectraForSequenceClassification, ElectraForTokenClassification, MegatronBertModel, MegatronBertForPreTraining, MegatronBertForNextSentencePrediction, MegatronBertForMaskedLM, MegatronBertForCausalLM, MegatronBertForMultipleChoice, MegatronBertForQuestionAnswering, MegatronBertForSequenceClassification, MegatronBertForTokenClassification, GPT2Model, GPT2LMHeadModel, GPT2LMHeadModel, GPT2ForSequenceClassification, GPT2ForTokenClassification, GPTJModel, GPTJForCausalLM, GPTJForQuestionAnswering, GPTJForSequenceClassification, GPTNeoModel, GPTNeoForCausalLM, GPTNeoForSequenceClassification, T5Model, T5ForConditionalGeneration, T5ForConditionalGeneration, RobertaModel, RobertaForMaskedLM, RobertaForMaskedLM, RobertaForCausalLM, RobertaForMultipleChoice, RobertaForQuestionAnswering, RobertaForSequenceClassification, RobertaForTokenClassification, GPT2DoubleHeadsModel`

@michaelbenayoun 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Merging with offline approval from @LysandreJik
[CI] switching CI to pytorch-1.11
Author: stas00
Pull request: no
Labels: name: Testing

Heads up: pytorch-1.11 will be released tomorrow

@LysandreJik , I recommend we switch our CI to it once it's released - note that nightly is 1.12-tobe so should remain as it is now. 

I have already requested torch-scatter binary wheels for 1.11 which will be needed to switch to : https://github.com/rusty1s/pytorch_scatter/issues/276

we can use this issue to track if anything else is needed.

Comments:
@LysandreJik, `pip install torch-scatter -f https://data.pyg.org/whl/torch-1.11.0+cu113.html` is already avaialble, 

`pip install torch-scatter -f https://data.pyg.org/whl/torch-1.11.0+cu115.html` should be available on Friday.

So no show-stoppers to switch to pt-1.11 now as soon as it's released.

At your convenience of course. I was just trying to make sure that all components are ready.

CI was switched to PyTorch 1.11 on the release date, `pytorch-scatter` was updated with the link you shared above. Thank you, for helping track it!
Configurable Relative Position Max. Distance
Author: agemagician
Pull request: yes

# What does this PR do?

The relative attention on the T5 model can only configured with the number of buckets.
However, it has fixed max distance and can't be configured.

The current fixed max distance is working fine for text modality, but for much longer sequence use-cases, it has to be adjusted.

This pull request make it possible to configure the max distance in the relative attention.

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@patrickvonplaten @sgugger  @patil-suraj @LysandreJik



Comments:
_The documentation is not available anymore as the PR was closed or merged._
[Fix doc example] Fix first example for the custom_datasets tutorial
Author: MarkusSagen
Pull request: yes

# What does this PR do?
- Example code for the sequence classification in Tensorflow (`custom_datasets` tutorial) had spelling mistake and running the example would not work.
- Changed the variable names to be consistent with the two other TF examples and for the example code to work

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Use templates
Author: sgugger
Pull request: yes

# What does this PR do?

This PR switches to use the templates in `doc-builder` for the doc building jobs.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Is `.github/workflows/build_documentation.yml` handled in a different PR ?  because it is deleted by this PR doesnt have its replacement

Are you sure you looked at the whole diff? It's just modified by the PR, not deleted.
Want to push a branch for a PR that fixes Deberta classification dropout and armonizes the dropout class, but have no permissions.
Author: avacaondata
Pull request: no

Hi, I'm trying to push a new branch that fixes Deberta classification dropout by adding cls_dropout to DebertaConfiguration. In this branch I've also changed the Dropout class of DebertaForTokenClassification to StableDropout, to use the same type of dropout as the other classes. 

However, I am not able to perform this operation even this is a new branch and wouldn't break anything in master or other branches... @LysandreJik  @vanpelt @patrickvonplaten How can I do it??

Comments:
Hi @alexvaca0 👋  From your message I can't confirm whether you're doing this or not, so here it goes: [we expect the contributions](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests) to come from *branches on a fork*, from which you can open a PR to the main repo. Have you tried doing this? :)

Hey @gante thank you very much for your answer :) I didn't know contributions needed to come from a fork! I'll do that then ! thank you ❤️ 

Awesome, glad I could help! Closing this issue :)
Change unpacking of TF layoutlm inputs to use decorator
Author: vumichien
Pull request: yes

# What does this PR do?

As part of #16051 this PR changes the unpacking of the inputs of TF layoutlm to use the decorator instead.

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
@Rocketknight1 @gante 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Yah, I agree Rag's test is suppper slow :( . Anyway, let me run all the tests again and I will commit when they are done

> Yah, I agree Rag's test is suppper slow :( . Anyway, let me run all the tests again and I will commit when they are done

I'm assuming all tests have passed, since you've made a commit -- is this correct? :D

Yeah, all the test have passed :D
Framework split
Author: sgugger
Pull request: yes

# What does this PR do?

This PR prepares the Transformers repo for the new syntax for framework-specific content. It should be merged once https://github.com/huggingface/doc-builder/pull/130 and https://github.com/huggingface/doc-builder/pull/63 are merged.

Note to @stevhliu : This only concerns existing code with switches, not text (except in the cases where the text dependent on PT vs TF was just before the code). It's probably worth doing a pass on the code to see what paragraphs could be split in framework specific blocks once this PR is merged. For instance, all task tutorials could have the whole fine-tuning in two such blocks.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_16030). All of your documentation changes will be reflected on that endpoint.
Added spanish translation of quicktour.mdx
Author: duedme
Pull request: yes

# Translation of quicktour.mdx into Spanish

I made the translation of quicktour.mdx into Spanish (fixes #15947). The document is located in the transformers/docs/source_es folder.

@omarespejel 

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks, @Duedme! LGTM!
Fix some Flax models' `hidden_states`
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix some Flax models where the last element in `hidden_states` is different between PT/Flax version.

## More context

Some models have 

```
last_hidden_state = self.layer_norm(last_hidden_state)
```

In Pytorch version, the returned `hidden_states` have this `last_hidden_state` (after layer norm) as the last element.
In Flax version, the last element of the returned `hidden_states` is the one before the layer norm.

This PR fixes this inconsistency (by using the PyTorch logic).



Comments:
_The documentation is not available anymore as the PR was closed or merged._