noamaanMulla-03/transformers-issues huggingface dataset:
Replicating SQuAD results on T5
Author: lucky-bai
Pull request: no

### System Info

Hi, I'm trying to replicate the SQuAD experiment in the [T5 paper](https://arxiv.org/abs/1910.10683). I'm following the paper's recommended hyperparameters for finetuning:

* AdaFactor optimizer
* Batch size 128 (I'm doing 16 per GPU on 8xRTX 3090 GPUs)
* 2^18 steps for fine-tuning (which is around 300 epochs)
* Max sequence length 512
* Learning rate 0.001

I'm running the following:

```run_seq2seq_qa.py --model_name_or_path t5-base --dataset_name squad --context_column context --question_column question --answer_column answers --do_train --do_eval --per_device_train_batch_size 16 --optim adafactor --learning_rate 0.001 --num_train_epochs 300 --evaluation_strategy epoch --max_seq_length 512 --predict_with_generate --output_dir /tmp/t5_squad/ --overwrite_output_dir```

After 4 epochs, the validation Exact Match score is 79.054 and F1 is 86.895. After 4 epochs, the model starts to overfit and the performance decreases. However, the paper reports 85.44 EM and 92.08 F1 score on T5-base (Table 14).

Has anyone been able to reproduce the official paper results or am I missing anything?

### Who can help?

@patrickvonplaten

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

See above

### Expected behavior

Should get around 85.44 EM and 92.08 F1 score on this task.

Comments:
Please use the [forums](https://discuss.huggingface.co/) for questions like this as we keep issues for bugs and feature requests only.
Note that we did not try to replicate the result of this paper with this script :-)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Use raw string for regex in tokenization_t5_fast.py
Author: odashi
Pull request: yes

# What does this PR do?

This change replaces the regex pattern written in a Unicode string to a raw string, to suppress `DeprecationWarning`/`SyntaxError` around the pattern.


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger and @raghavanone who organized the original code.


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add UperNet
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds the classic [UperNet](https://arxiv.org/abs/1807.10221) framework to Transformers.

Many papers that introduce a new vision backbone, such as BEiT, ConvNeXt, Swin,... benchmark their model on downstream tasks such as semantic segmentation and object detection. All of these papers use the UperNet framework (introduced in 2018) when evaluating their backbone on semantic segmentation.

Hence, this PR implements this framework, making use of the new [AutoBackbone API](#20229) to make the following possible:

```
from transformers import SwinConfig, UperNetConfig, UperNetForSemanticSegmentation

backbone_config = SwinConfig(out_features=["stage1", "stage2", "stage3", "stage4"])

config = UperNetConfig(backbone_config=backbone_config)
model = UperNetForSemanticSegmentation(config)
```

In the code above, we're instantiating the UperNet framework with Swin Transformer as backbone. The code looks equivalent for another backbone, like ConvNeXt:
```
from transformers import ConvNextBackbone, UperNetConfig, UperNetForSemanticSegmentation

backbone_config = ConvNextBackbone(out_features=["stage1", "stage2", "stage3", "stage4"])

config = UperNetConfig(backbone_config=backbone_config)
model = UperNetForSemanticSegmentation(config)
```

To do:

- [ ] looking into supporting `from_pretrained` of backbones => will be done in a follow-up PR
- [x] make sure UperNetImageProcessor does exact same preprocessing
- [x] make UperNetImageProcessor also take `segmentation_maps` as optional input
- [x] add image processor tests
- [x] convert all checkpoints + update organization
- [x] fix integration tests

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for the review, I'm waiting for the authors to respond regarding the creation of an organization on the hub.
[VideoMAE] Fix docstring
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR fixes a docstring for VideoMAE (the model doesn't have a CLS token).

Fixes #21016 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Cc @ydshieh seems like the CI has only run 7 checks. Any idea why?

Because it doesn't touch any code. Tests are only run for code changes.

Because it doesn't touch any code. Tests are only run for code changes.
QuestionAnsweringPipeline top_k returns single result
Author: henrique-b
Pull request: no

### System Info

- `transformers` version: 4.24.0
- Platform: Linux-5.15.0-1025-gcp-x86_64-with-glibc2.31
- Python version: 3.9.15
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.9.0+cu111 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes: tesla T4
- Using distributed or parallel set-up in script?: No


### Who can help?

@Narsil

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

When using a QuestionAnsweringPipeline with the the `top_k` parameter set to a number greater than 1, the model can still return a single answer in the form of a dictionary. 

Example to reproduce bug:

```[python]
from transformers import AutoModelForQuestionAnswering, AutoTokenizer, QuestionAnsweringPipeline

pipeline = QuestionAnsweringPipeline(
    model=AutoModelForQuestionAnswering.from_pretrained("distilbert-base-cased-distilled-squad"),
    tokenizer=AutoTokenizer.from_pretrained("distilbert-base-cased-distilled-squad")
)

pipeline([{
    "context": " 1 ",
    "question": "What is Anne's age?"
}], top_k=10)
``` 

### Expected behavior

When the `top_k` parameter, I would expect that the call to the model returns a list containing the best predictions up to the tenth, when possible. If the model only outputs one answer, I would expect this answer to be within a list. 

When there are no possible answer, the returned value is an empty list. When there are multiple answers, the returned value is also a list. Outputting a dictionary creates an edge case that needs to be handled when, for example, iterating over the outputs of the model 

Comments:
You are entirely correct to wish for it.

The current behavior is linked to our commitment to not break things. 

If you actually check out the code, you'll see this is an exception because there's only 1 returned value (Because only 1 return value is possible).
https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/question_answering.py#L596-L597
There's actually another one: https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/question_answering.py#L391-L392

Since we're not breaking things, and this pipeline was written a long time ago, it was not necessarily aligned with other pipelines. So these quircks are unfortunately necessary.

This is the one thing I would like to modify in V5 (cleanup pipelines return types to make them extremely consistent).

I hope you understand the current state of things. Not sure if there's anything to be done about it.
[LongT5] Remove duplicate encoder_attention_mask default value check
Author: guillaume-be
Pull request: yes

# What does this PR do?

This PR performs a minor code clean-up by removing duplicate code in the LongT5 stack. 
Currently, if the stack is a decoder block with `encoder_hidden_states` provided, the check for the existence of `encoder_attention_mask` is done twice:

1. https://github.com/huggingface/transformers/blob/c8f35a9ce37bd03f37fcf8336172bdcbe7ffc86a/src/transformers/models/longt5/modeling_longt5.py#L1452
2. https://github.com/huggingface/transformers/blob/c8f35a9ce37bd03f37fcf8336172bdcbe7ffc86a/src/transformers/models/longt5/modeling_longt5.py#L1479

Because the conditions on the second check are identical to that of the first check (`self.is_decoder is True`, `encoder_attention_mask is None`, `encoder_hidden_states is not None`), the second check will never be True since  `encoder_hidden_states` was updated during the second check.

This PR proposes removing the first check and only use the second when the extended encoder attention mask is set anyway.

## Who can review?
 @ArthurZucker , @younesbelkada 


Comments:
_The documentation is not available anymore as the PR was closed or merged._
remove flax from `documentation_tests.txt`
Author: ydshieh
Pull request: yes

# What does this PR do?

#21009 added `src/transformers/generation/flax_utils.py` to `documentation_tests.txt`, but CI image doesn't have `jax/flax` installed. The whole doctest suite failed, and reported 0 failure.

This PR removes this file from `documentation_tests.txt`. The CI image used is the same as the scheduled CI, and it's intended not to have `jax/flax` in this image.

We can decide if to use a separate image though.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Ah, this explains why the previous example had several issues 😅 

@ydshieh We don't plan to add FLAX to the doctests, correct?

@gante Not a no from me. I think it's good to make sure the examples work (this is in the range of maintenance mode 😄  ). But I would like to have a yes from @sgugger and @LysandreJik before working on it.

Not worth any work right now given the usage IMO. There is plenty to do with more priority :-)
Fix `RealmModelIntegrationTest.test_inference_open_qa`
Author: ydshieh
Pull request: yes

# What does this PR do?

The test `RealmModelIntegrationTest::test_inference_open_qa` fails since my PR #21000. This integration test creates a config manually by `config = RealmConfig()` and passes it to `from_pretrained`, therefore it doesn't have the attribute `searcher_seq_len` (which is removed in #21000).

Using `from_pretrained` without specifying `config` will load from the Hub checkpoint which has `searcher_seq_len` in the config (loaded via `super().__init__(..., **kwargs)`), and fix the test.



Comments:
_The documentation is not available anymore as the PR was closed or merged._
Added clefourrier as ref point for graph models in bug reports
Author: clefourrier
Pull request: yes

# What does this PR do?
Added myself as entry point for graph models in issue doc.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Also, I missed this file
```
.github/PULL_REQUEST_TEMPLATE.md
```
if you think it is relevant.

Add @sgugger for final review as I never changed these files before, and this is also more administrative decision :-)

Did not see the comment section in PULL_REQUEST_TEMPLATE when I opened them in browser view - edited!

Edit: However, not sure what I could add to `feature-request`?

> Did not see the comment section in PULL_REQUEST_TEMPLATE when I opened them in browser view - edited!
> 
> Edit: However, not sure what I could add to `feature-request`?

My bad, my brain is not completely recovered from all the `drink` I had last week.
[GIT] Fix training
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR ensures that GIT can be properly fine-tuned. As GIT is a causal, GPT-like model that is also conditioned on CLIP-embedded image patches, one needs to only compute a loss on the predicted text tokens.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks, fixed now!
Feature: Adding support for MultiWorkerMirroredStrategy in TensorFlow Training Args
Author: Lokiiiiii
Pull request: yes

# What does this PR do?
Adding support for MultiWorkerMirroredStrategy in TensorFlow Training Args. This is an existing stable Distribution strategy provided by TensorFlow.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@sgugger @jplu




Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20682). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[layoutlmv3] SER and RE task combined into one model
Author: githublsk
Pull request: no

### System Info


datasets==1.15.1
transformers==4.12.5
seqeval==1.2.2
deepspeed==0.5.7
tensorboard==2.7.0
seqeval==1.2.2
sentencepiece
timm==0.4.12
Pillow
einops
textdistance
shapely

### Who can help?

@NielsRogge

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

combine SER and RE task into one model using layoutlmv3

### Expected behavior

Hi @NielsRogge,
Now we want to train SER and RE task using layoutlmv3, but these two models is more heavy for deploy with tensorrt, so to solve this problem, if we combine two tasks into one model, the performance for SER and RE will decrease more compared to the seprated two models?  we have no more experience for it, do you give us some advice? thank you very much.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I think it's totally doable to share the Transformer encoder and use 2 separate heads. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Closing this issue as it seems resolved. Feel free to reopen.
Add SpA-Former
Author: kanpuriyanawab
Pull request: no
Labels: name: New model

### Model description

I would like to add [SpA-Former](https://arxiv.org/abs/2206.10910) model to the Transformers.

It is an end-to-end transformer to recover a shadow-free image from a single shaded image. Unlike traditional methods that require two steps for shadow detection and then shadow removal, the SpA-Former unifies these steps into one, which is a one-stage network capable of directly learning the mapping function between shadows and no shadows, it does not require a separate shadow detection. Thus, SpA-former is adaptable to real image de-shadowing for shadows projected on different semantic regions. SpA-Former consists of transformer layer and a series of joint Fourier transform residual blocks and two-wheel joint spatial attention. The network in this paper is able to handle the task while achieving a very fast processing efficiency. 

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

[Link](https://github.com/zhangbaijin/SpA-Former-shadow-removal) to Model Repo 
[Link](https://arxiv.org/abs/2206.10910) to Paper 

Comments:
@NielsRogge Would it be a valuable contribution to HuggingFace?

Sure this would be valuable! Let me know if you need any help
opt-13b checkpoint missing final_layer_norm weights in pretrained checkpoint
Author: zw123han
Pull request: no

### System Info

Hello. Apologies if this has been brought up before, but it seems at least the flax version of opt-13b when using from_pretrained is missing weights for:

- model/decoder/final_layer_norm/bias
- model/decoder/final_layer_norm/scale

Every other version of the model I've tested didn't have this issue. You can see the message below.
![image](https://user-images.githubusercontent.com/67124639/209904124-1aed006f-d20b-4cd8-9602-fe90f8da5d71.png)


### Who can help?

@ArthurZucker 

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I'm using a modified version of `run_clm_flax.py` under `examples/flax/language-modelling`, although my modification below should not make an impact and the default script should have the same issue. 

`config = AutoConfig.from_pretrained(
            'facebook/opt-13b',
        )
`

`model, params = FlaxAutoModelForCausalLM.from_pretrained(
            'facebook/opt-13b',
            config=config,
            seed=42,
            _do_init = False, 
        )`

`params = model.init_weights(model.key, model.input_shape, params).unfreeze()` # inside a jit function
        

### Expected behavior

You will notice the message above which shows missing weights for only the last layer of this specific model. 

Comments:
Hey, there is indeed an issue with the `FlaxWeights`, if you try using `from_pt = True` you should have the correct layers loaded. There are actually 2 different checkpoints online, one is sharded and the other is not. This is pretty strange 😓 

Hi Arthur, thanks for the help! Managed to convert PyTorch weights and all of them are there!

Although, for anyone else using `_do_init = False` in `from_pretrained`, I had to convert and save to Flax weights first to a local dir as the `from_pt = True` flag will attempt to internally call `model.params`.
Support for Monotonic Mulithead Attention based Simultaneous Speech-to-text Translation
Author: beomseok-lee
Pull request: no
Labels: name: New model

# 🌟 New model addition
Simultaneous Speech-to-text Translation using Monotonic Multihead Attention(MMA).
I am wondering if anybody is working on implementing this model for now.
However, I am worried that if this model is going to be supported by Hugging Face systems, since inference works in a particular way using frameworks like [SimulEval](https://github.com/facebookresearch/SimulEval) to simulate streaming input which may not be compatible with current Hugging Face's inference system?

## Model description
[MMA(Ma et al., 2019)](https://arxiv.org/abs/1909.12406) has been used to handle streaming text/speech inputs mostly for translation, where MMA extends the monotonic attention mechanism to multihead.

<!-- Important information -->

## Open source status
* [x] the model implementation is available: [Fairseq Implementation is available here](https://github.com/pytorch/fairseq/blob/fcca32258c8e8bcc9f9890bf4714fa2f96b6b3e1/examples/simultaneous_translation/models/convtransformer_simul_trans.py#L29~#L63)
* [ ] the model weights are available: (give details)
* [ ] who are the authors: (mention them, if possible by @gh-username) : Xutai Ma(@xutaima), Juan Pino, James Cross, Liezl Puzon, Jiatao Gu

Inference framework : [Facebook Research SimulEval](https://github.com/facebookresearch/SimulEval)

Comments:
@beomseok-lee can I work on this issue?
Optimize inference only mode memory if ipex is used
Author: sywangyi
Pull request: yes

Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>

# What does this PR do?

optimize the memory if used ipex for inference.

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?


Library:

- trainer: @sgugger


Comments:
@sgugger  @jianan-gu @yao-matrix please review

_The documentation is not available anymore as the PR was closed or merged._