noamaanMulla-03/transformers-issues huggingface dataset:
Type hints MCTCT
Author: rchan26
Pull request: yes

Hi @Rocketknight1: this PR looks to add type hints to MCTCT models and addresses #16059.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Ah, careful! `attention_mask` and `head_mask` **were** optional in the main model methods, but not in the encoder. The encoder mostly doesn't have `Optional` arguments, but the other classes do. The changes to the main model methods is why you're getting that error now!

This is much easier to see if you look at the 'files changed' interface on GitHub - we want this PR to only add type annotations, but not change any default arguments.

I see - my apologies! Will make some changes now to fix this :)
TF port of ESM
Author: Rocketknight1
Pull request: yes

Working out the last few issues now! Models <3B parameters have been ported already, larger models will need to wait for #19124. 

This PR also includes fixes for a couple of issues in the original PyTorch ESM.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Pipeline tests are failing because the model has no SEP token and doesn't work with multiple sequences. Working on it!

There's one final test remaining that's failing because of some arcane issue in the code that generates data batches for the pipeline. I'm trying to figure it out!

Tests are green, and #19124 has been merged! Going to use it to upload the remaining checkpoints and then merge this.
Getting the heat map out of VILT (Figure 4 in the paper)
Author: Ngheissari
Pull request: no
Labels: name: Good First Issue

### Feature request

I would like to get the heatmap from ViLT (Visualizations of transportation plan of word patch alignment). 

### Motivation

It is useful for debug and also might be useful for other applications. It is shown in Figure 4 in the paper: https://arxiv.org/pdf/2102.03334.pdf

### Your contribution

NA

Comments:
Hi,

Yes it's totally possible to re-create that. Basically it comes down to translating [this script](https://github.com/dandelin/ViLT/blob/master/demo.py) to a Gradio demo.

I'm marking this as "good first issue" as it seems fairly straightforward.

Hi, I would like to work on it

@NielsRogge Is this issue still open ? I would to like to work on this. Can you assign this issue to me ?


@NielsRogge , is this issue still open I would like to contribute to this 

Yeah, I guess. I think you guys can go ahead and open PR.

> @NielsRogge , is this issue still open I would like to contribute to this

Hey Rajath ! Would love to work on this issue with you. Do you mind working with me on this? Kinda new to open source contribution. 


Is anyone still working on this? I can help

@NielsRogge is this supposed to be implemented as a method on all vilts or a function that takes a vilt model as input and launches the gradio demo? 

You can just implement a Gradio demo and host it on https://huggingface.co/spaces.

> You can just implement a Gradio demo and host it on https://huggingface.co/spaces.

I've made one https://huggingface.co/spaces/MikailDuzenli/vilt_demo , I just implemented the demo of the model itself for the moment but I'm trying to add the heatmap (help is welcome).

> > You can just implement a Gradio demo and host it on https://huggingface.co/spaces.
> 
> I've made one https://huggingface.co/spaces/MikailDuzenli/vilt_demo , I just implemented the demo of the model itself for the moment but I'm trying to add the heatmap (help is welcome).

I could help

Very cool demo @MikailINTech! Awesome work. Final step is indeed visualizing the heatmap

> Very cool demo @MikailINTech! Awesome work. Final step is indeed visualizing the heatmap

Thank you ! Just finished adding the heatmap. Is there a way I can have this issue mark as resolved ?

Really cool! Although I'd also include the entire image in the result (not just the heat map), to compare.

Then I'll close this issue!

Thanks @NielsRogge for the suggestion, now one can see the image and the heatmap. I hope that it's what @Ngheissari was looking for 

Awesome! Closing this issue.

I tweeted about it here :) https://twitter.com/NielsRogge/status/1580246704011370496
Add pillow to layoutlmv3 example requirements.txt
Author: Spacefish
Pull request: yes

Adds required pillow / PIL library to the layoutlmv3 training example, as it´s used to load the images during training.

@sgugger, @patil-suraj
tagging you as i don´t know who is reponsible for that example.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Tokenizer not loaded for image-to-text pipelines when specified
Author: davidmezzetti
Pull request: no

### System Info

Transformers 4.23.1

### Who can help?

@Narsil 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import pipeline
nlp = pipeline("image-to-text", model="ydshieh/vit-gpt2-coco-en", tokenizer="ydshieh/vit-gpt2-coco-en")
nlp("image.jpg")
```

Results in:

```
  File "transformers/pipelines/image_to_text.py", line 89, in postprocess
    "generated_text": self.tokenizer.decode(
AttributeError: 'str' object has no attribute 'decode'
```

### Expected behavior

Caption to be returned.

It appears something around this line is being tripped up - https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/__init__.py#L733

Comments:
cc @Narsil 

If that is an unexpected issue, I can debug it and try to fix. 

Hi @davidmezzetti ,

The "string" is not resolved in `pipeline` when both `model` and `tokenizer` are sent.
It should work when you send only 1.

You could definitely create a PR to try and resolve `tokenizer` too in that situation.
It's tricky magic code so be careful.

Thanks @narsil. In reviewing the way I'm calling pipelines, there is no reason to pass both a `model` and `tokenizer` as they are the same.

It appears that this line: https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/__init__.py#L737 could be updated to also check if the tokenizer is a (str, tuple) for multi models. But it seems like a highly niche/possibly unnecessary use case. 

`pipeline` is MAGICAL by nature, I'm not against adding even more magic to it. As long as the actual pipeline classes, stay much more down to earth and less magical. 

Magic is super nice when it works, but much harder to work with/evolve when it doesn't :)

Sounds good, I'll go ahead and close this issue.
fix warnings in deberta
Author: sanderland
Pull request: yes

# What does this PR do?

In recent torch versions, the following warning is thrown on the deberta code:
```
 UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  p2c_att = torch.matmul(key_layer, torch.tensor(pos_query_layer.transpose(-1, -2), dtype=key_layer.dtype))
```
This PR fixes the warnings by using .to(dtype=)  rather than tensor(...,dtype=)




Comments:
_The documentation is not available anymore as the PR was closed or merged._

@LysandreJik any thoughts?
[Doctest] Add configuration_transfo_xl.py
Author: thliang01
Pull request: yes

# What does this PR do?

Add `configuration_transfo_xl.py` to `utils/documentation_tests.txt` for doctest.

Based on issue  #19487

@ydshieh could you please take a look at it?
Thanks =)


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@thliang01 Could you try to resolve the conflict :-) thank you 🙏 

Hi @thliang01 I tried to fix the conflict and make it clean. It should work now - I will merge once the CI are all good. Thank you again for your contribution 👍 💯 !
object-detection instead of object_detection
Author: Spacefish
Pull request: yes

The Object detection Sample in the README.md does not work out of the box,
as it tries to initialize a pipeline `object_detection` however it is called `object-detection`

Not sure if this is intended such that a new user has to overcome a minimal sanity check?! 🤣 

@sgugger

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Update decision transformers to gym 0.26
Author: RedTachyon
Pull request: no

### Feature request

We recently published a [new release of gym](https://github.com/openai/gym/releases/tag/0.26.0), which carries with it a bunch of breaking changes. 
However, this is the last of the API changes, and it will be stable going forward. So it would be great to update the decision transformers to be compatible with that.

### Motivation

I'd say there are two main reasons to switch to the new API that goes with 0.26:
- The new API "makes sense"- we're still preparing a proper writeup of the rationale behind each decision, but they were deliberately made to support good research, flexibility and reproducibility. 
- It will be supported in the future - we have many exciting features on the horizon (e.g. hardware-accelerated environments), which will be predicated on using the new API

### Your contribution

I'll be happy to help with the whole process, including contributing the PR. 

From what I can see, most of the code in transformers is rather self-contained, so it would mainly be the the `run_decision_transformer.py` example that needs updating, and then potentially other resources about decision transformers (like the blog), which would be separate PRs naturally

The biggest question would be how you want to handle versioning. My intuition is that it'd be best to update to gym 0.26 together with some "significant" version of transformers, like `4.22 -> 4.23` (or later, depending on how long it takes). 

Comments:
cc @edbeeching 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
BertTokenizer slowly on latest versions
Author: Piecer-plc
Pull request: no
Labels: name: bug

### System Info

Hi, I test the execution time of my program when adopt different transformers version.
In my program, when the transformers version was **4.21.3**, the execution time was **7.12s**, but when I changed the lightgbm to the **4.3.0**, the execution tim was **2.79s**.
I record the system info on different versions.
[4.21.3.txt](https://github.com/huggingface/transformers/files/9532526/4.21.3.txt)
[4.10.0.txt](https://github.com/huggingface/transformers/files/9532528/4.10.0.txt)
[4.3.0.txt](https://github.com/huggingface/transformers/files/9532527/4.3.0.txt)

### Who can help?

@LysandreJik 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1: Code
```python

from transformers import BertTokenizer
import pandas as pd
import time
start = time.time()
train_df = pd.read_csv('train.csv')
train_df.head()
tweets = train_df['text'].values
tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased')
def encode_sentence(s):
    tokens = list(tokenizer.tokenize(s))
    tokens.append('[SEP]')
    return tokenizer.convert_tokens_to_ids(tokens)
max_len = 50
x_train = []
for tweet in tweets:
    vec = encode_sentence(tweet)
    x_train.append(vec[:max_len] + [0] * (max_len - len(vec)))
end = time.time()
print("Time:", end-start)

```
2: Dataset
[train.csv](https://github.com/huggingface/transformers/files/9532537/train.csv)


### Expected behavior

The execution time are same on different versions, or the latest version is better than older versions.
|Version|Execution time|
|--|--|
|4.21.3|7.12s|
|4.10.0|8.85s|
|4.3.0|2.79s|

Comments:
cc @SaulLu 

Hi @PerformanceDetect ,

Thanks for sharing this benchmark with us! Indeed it would be interesting to find out what addition caused this slowdown, do you think you would have some time to investigate further?

Otherwise, if your usage is speed sensitive, I recommend you to use the fast version of the tokenizer :hugs: 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Pooled output from DeBERTa(v2) model
Author: IgorPidik
Pull request: no

### Feature request

Add a context pooler and include the `pooler_output` in the result of the `forward` method of the `(TF)Deberta(V2)Model`.

Changes required:
* introduce `add_pooling_layer` flag in the constructor
* add pooler
* change return type from `BaseModelOutput` to `BaseModelOutputWithPooling`
* update heads which currently do their own context pooling

### Motivation

* Simplification of use cases where the model is used to generate context embeddings (whether the embeddings being the end goal or only used as an input to additional layers in a custom model)
* Currently multiple heads require the pooled output and hence it is re-implemented in multiple places. The complexity of the heads and the redundancy can be decreased by moving this operation to the base model
* Consistency with BERT and RoBERTa implementations which already operate in this fashion. Additionally, this consistency can simplify use cases where the user works with context embeddings and wants to experiment with different models. E.g.: BERT and RoBERTa can be easily exchanged. On the other hand, swapping in DeBERTa requires the user to handle the context pooler

### Your contribution

I implemented the necessary changes in my fork and can submit a PR

Comments:
Thanks for your suggestion! This will sadly be a breaking change as it will change the way the weights are named, and thus break compatibility with all DeBERTa models on the hub. So even if your suggestion makes a ton of sense, I don't think we will be able to implement it.

@sgugger thank you for your reply! I see why the implementation of the heads should not be altered, nevertheless, wouldn't it still make sense to at least add the pooler to the base model (the deberta model without any head)? Motivation being the first and the last points from the list above.



We have plenty of other models (like ELECTRA) where the pooler is not implemented. As a general rule of thumb, it depends on whether it was in the pretraining objective or not.

I understand and thank you! I will close the issue then.
[Doctest] Add `configuration_data2vec_text.py`
Author: daspartho
Pull request: yes

Add `configuration_data2vec_text.py` to `utils/documentation_tests.txt` for doctest.

Based on issue #19487

@ydshieh could you please check it?
Thanks :)

Comments:
@ydshieh made the suggested changes =)

_The documentation is not available anymore as the PR was closed or merged._
Fixed the docstring and type hint for forced_decoder_ids option in Ge…
Author: koreyou
Pull request: yes

# What does this PR do?

This PR fixes #19602 where the docstring and type hint for forced_decoder_ids option in GenerationMixin.generate were inconsistent with the actual implementation.

Fixes #19602

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@gante has suggested me to send a PR for the issue #19602. 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@gante Thanks for the comment! I've taken in your suggestion in the new commit. Please proceed with the merge if it is looking good.

@koreyou Awesome, thank you for the changes! I will merge as soon as CI turns to green :)
Add normalize to image transforms module
Author: amyeroberts
Pull request: yes

# What does this PR do?

Adds `normalize` to the image transforms modules, as well as a helper utility function `get_channel_dimension_axis`. 

* `normalize`: performs equivalent normalization of an image as previous feature extractors
* `get_channel_dimension_axis`: Helper function which returns which axis number the channel dimension is on.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[Doctest] Add `configuration_data2vec_audio.py`
Author: daspartho
Pull request: yes

Add `configuration_data2vec_audio.py` to `utils/documentation_tests.txt` for doctest.

Based on issue #19487

@ydshieh could you please take a look at it?
Thanks :)

Comments:
Could you pull the latest `main` from the remote (which include your PR with other data2vec config files), and rebase your PR branch on the new `main` 🙏 . We need to fix the conflict changes

@ydshieh rebased the branch, it should resolve the conflict :)

_The documentation is not available anymore as the PR was closed or merged._
Removed XLMModel inheritance from FlaubertModel(torch+tf)
Author: D3xter1922
Pull request: yes

# What does this PR do?
related to #19303

Removed XLMModel inheritance from FlaubertModel for pytorch and tensorflow

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@sgugger 


Comments:
Hi @sgugger I am failing some tests, can you please help me with this?

_The documentation is not available anymore as the PR was closed or merged._

@sgugger Hi. esm model test is failing but I am not able to figure out why. Can you please help?


Arf, sorry I misled you. The copied from came from the code of XLM (they are copied from BERT actually) and you need to keep them. Sorry about that!

Thank you for your help!
num_proc in dataloader affect F1 score in squad
Author: Yang-YiFan
Pull request: no

### System Info

transformers 4.18.0 (on CPU)
python 3.9

### Who can help?

@sgugger 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

In the example question answering script, `cd examples/pytorch/question-answering/`

Running the below two cmds will yield different F1 score report (even though their difference is just num workers in dataloader):
1. `python run_qa_no_trainer.py --model_name_or_path csarron/bert-base-uncased-squad-v1 --dataset_name squad --max_seq_length 384 --doc_stride 128 --num_train_epochs 0 --preprocessing_num_workers 1 --output_dir ~/tmp/debug_squad`

2. `python run_qa_no_trainer.py --model_name_or_path csarron/bert-base-uncased-squad-v1 --dataset_name squad --max_seq_length 384 --doc_stride 128 --num_train_epochs 0 --preprocessing_num_workers 4 --output_dir ~/tmp/debug_squad`

### Expected behavior

1. `Evaluation metrics: {'exact_match': 80.90823084200568, 'f1': 88.22754061399627}`
2. `Evaluation metrics: {'exact_match': 76.51844843897824, 'f1': 83.40809222646291}`
They have different results on squad.

Comments:
Hi there! Could you make your model public? I cannot reproduce this on [distilbert-base-cased-distilled-squad](https://huggingface.co/distilbert-base-cased-distilled-squad) (I get the same scores for the two commands).

Hi, I believe the one I'm using `csarron/bert-base-uncased-squad-v1` is public. I also tried `bert-large-uncased-whole-word-masking-finetuned-squad` and it's showing the same F1 score mismatch. Thanks!

Ah, just caught the problem in the logs. `preprocessing_num_workers` is the number of workers sent to `Dataset.map`. It should be left as 1 when using a fast tokenizer. When you change it, you change the way the dataset is preprocessed.

To change the number of workers in the dataloader, you should use `dataloader_num_workers`.

Thanks! The default value is still set to 4 [here](https://github.com/huggingface/transformers/blob/5fda1fbd4625e93d023fe02153ec4a05b26b16cc/examples/pytorch/question-answering/run_qa_no_trainer.py#L111), which should be 1 according to your finding.

Indeed. Do you want to make a PR to fix this?

Done.