noamaanMulla-03/transformers-issues huggingface dataset:
Add table transformer [v2]
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds [Table Transformer](https://github.com/microsoft/table-transformer) by Microsoft, as a separate model, rather than tweaking the existing DETR implementation.

Comments:
@sgugger I've removed the non-Table Transformer related stuff to #19644 ;)

_The documentation is not available anymore as the PR was closed or merged._

@sgugger PR is ready, feel free to approve :)
Seq2SeqTrainer predict with generate fails to run without labels
Author: IvanSedykh
Pull request: no

### System Info

- `transformers` version: 4.21.2
- Platform: macOS-12.6-arm64-arm-64bit
- Python version: 3.9.12
- Huggingface_hub version: 0.9.1
- PyTorch version (GPU?): 1.12.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@patrickvonplaten, please check.

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

This code fails.

```python3

import transformers
from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments, AutoModelForSeq2SeqLM, AutoTokenizer
from datasets import Dataset

model_name = "google/t5-efficient-tiny"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

ds = Dataset.from_dict({"text": ["Sample text to continue 1:", "Sample text to continue 2:"]})
ds = ds.map(lambda x: tokenizer(x["text"], truncation=True, padding="max_length", max_length=512), batched=True)

trainer_args = Seq2SeqTrainingArguments(
    "tmp_output",
    predict_with_generate=True,
)

trainer = Seq2SeqTrainer(
    model,
    args=trainer_args,
)
predict_output = trainer.predict(ds) # FAILS
```


```
ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds
```

### Expected behavior

It seems to me that Seq2SeqTrainer should output generation results. Therefore this class may be used for model inference.

Of course, in this case it is not possible to calculate metrics or loss, thus they should be None.

I know how to fix it with tiny change in Seq2SeqTrainer class, so I will create a PR later today.


Comments:
[Doctest] - Fixing doctest `configuration_pegasus.py`
Author: mukesh663
Pull request: yes

# What does this PR do?
Fixes #19487 
Added (with random weights) in `configuration_pegasus.py`.
Added `configuration_pegasus.py` in `documentation_tests.txt`

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?
@ydshieh 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19722). All of your documentation changes will be reflected on that endpoint.
fix seq2seqtrainer predict without labels
Author: IvanSedykh
Pull request: yes

# What does this PR do?

This PR allows to use Seq2SeqTrainer for prediction with generation on dataset without target sequence. So None loss and metrics are returned.

Technically, model.forward would be called only if we have 'labels'


Fixes #19714


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@patrickvonplaten, please check this PR.


Comments:
> This means the model will not be run if there are no labels. I don't think this fixes the issue as you wanted.

Thank for your reply.

In fact, in `prediction_step()` `model.generate` is still called, so trainer.predict would return generated_tokens, but would not compute loss.
See [here](https://github.com/huggingface/transformers/blob/dd523da577f3d1471d570f0bc388af55d026ce95/src/transformers/trainer_seq2seq.py#L198).

I have tried the new trainer by myself and the code snippet from the issue works properly.




_The documentation is not available anymore as the PR was closed or merged._
Add configuration_wav2vec2.py 
Author: juancopi81
Pull request: yes

Add configuration_wav2vec2.py to utils/documentation_tests.txt for doctest.

Based on issue: [19487](https://github.com/huggingface/transformers/issues/19487)

@ydshieh could you please check it?
Thanks :)


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you very much @ydshieh!!
Parameterize hidden layers for feature-extraction pipeline
Author: carschno
Pull request: no

### Feature request

In the literature, token embeddings for downstream tasks are generated from various combinations of hidden layers, e.g. by summing or concatenating the last four layers. [This image](http://jalammar.github.io/images/bert-feature-extraction-contextualized-embeddings.png) from [this blog entry](http://jalammar.github.io/illustrated-bert/) gives a few examples. Currently, the `feature-extraction` outline only extracts the last hidden state.

I propose an additional parameter that specifies the layers to be used would be nice, e.g. for summing up the last four layers:

    pipeline('feature-extraction', n_layers=4)

I suppose there are a few options to discuss, e.g. is a single `n` to imply a range like `[-n:]` sufficient, or should it also be possible to specify e.g. `[-4:-2]` and/or `[2:]`?

Furthermore, alternative aggregation methods could be defined, e.g.:

     pipeline('feature-extraction', n_layers=4, aggr=('sum'|'concat'|...))



### Motivation

I understand from issue https://github.com/huggingface/transformers/issues/4613 that the output of the `feature-extraction` pipeline comes from the last hidden layer. I think this is a reasonable default, but it would be good to have an easy way to compare results when using other layers too. 

### Your contribution

I have never worked with `transformers` source code, but I suppose I could find my way if I get a pointer to the relevant place(s).

Comments:
In general pipelines are kept simple for demo purposes. You can easily instantiate the processing class and model yourself for more complex behavior.


> In general pipelines are kept simple for demo purposes. You can easily instantiate the processing class and model yourself for more complex behavior.

Ok, I suppose my point of view would be that you might want to be able to demonstrate the same feature with summing several layers. But I understand it is a design decision that is a bit subjective.
fix the error which will make shape not match occur.
Author: novioleo
Pull request: yes

# What does this PR do?

When using PyTorch for model trace, the size of feature map will be inconsistent, resulting in failure to export successfully. The reason is that += is used in the size calculation code, and the object that happens to be added happens to be a shallow copy object, causing input_shape to change. You can solve this problem by declaring variables directly.

# Test script
```python3
from einops import repeat
from transformers import BertTokenizerFast,LayoutXLMTokenizerFast,LayoutLMv2ForSequenceClassification
import torch


## dummy inputs
dummy_input_ids = torch.LongTensor(torch.randint(low=0, high=1000, size=(2,256)))#.cuda()
box = [[48, 84, 73, 128]] * 256
dummy_bboxes = repeat(torch.LongTensor(box).unsqueeze(0), '1 b s-> 2 b s')
dummy_attention_mask = torch.LongTensor(torch.randint( low=0, high=1024, size=(2, 256) ))#.cuda()
dummy_imgs = torch.randn(2, 3, 448, 448)#.cuda()
dummy_token_ids = torch.zeros_like(dummy_attention_mask)

dummy_inputs = [
    dummy_input_ids,
    dummy_bboxes,
    dummy_imgs,
    dummy_attention_mask,
    dummy_token_ids
    ]
with torch.no_grad():

    model = LayoutLMv2ForSequenceClassification.from_pretrained("microsoft/layoutlmv2-base-uncased", torchscript=True,num_labels=30522)
    model.eval()
    traced_model = torch.jit.trace(func=model,
                                       strict=False,  
                                       example_inputs=dummy_inputs)
torch.jit.save(traced_model,'temp.pt')
model = torch.jit.load('temp.pt')
model.eval()
with torch.no_grad():
    result = model(*dummy_inputs)
print(result)
```

**If the script is run directly before the code is modified, the feature map size will be inconsistent**

# Who can review?
@LysandreJik


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18789). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Adding config files for convnext
Author: soma2000-lang
Pull request: yes

# What does this PR do?
Adding config files for convnext
Based on issue: https://github.com/huggingface/transformers/issues/19487

@ydshieh could you please check it?
Thanks :)

Comments:
Hi @soma2000-lang Could you run `make style` to see what's wrong with the CI and fix the potential issues 🙏 Thanks

Ok

On Tue, 18 Oct 2022, 18:41 Yih-Dar, ***@***.***> wrote:

> Hi @soma2000-lang <https://github.com/soma2000-lang> Could you run make
> style to see what's wrong with the CI and fix the potential issues 🙏
> Thanks
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/pull/19717#issuecomment-1282364070>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ANLS36I3ILRYJW4KW5DXX43WD2OZBANCNFSM6AAAAAARIBH7TE>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


![image](https://user-images.githubusercontent.com/56045049/196441022-4172d54e-bb7b-465a-b38a-64763d486f03.png)
@ydshieh After running make style I am getting the above.I am unable to understand what cold have been the problem

OK. First thing to resolve: your PR also changes the following file (and also in a strange way)
```
src/transformers/models/clip/configuration_clip.py
```

The goal of this PR is to work on convnext. We should not change the above file. Could you try to make this PR clean by not doing changes to CLIP config filei 🙏 ?

@ydshieh My last pr was about adding config files for the src/transformers/models/clip/configuration_clip.py thats  why changes were also reflected in this pr .Thats why I deleted the src/transformers/models/clip/configuration_clip.py file.

You can't just delete that file, as it means the file will be deleted when the PR is merged to the library.
You should instead to revert that file to the original state.

_The documentation is not available anymore as the PR was closed or merged._
Use TF framework
Author: ydshieh
Pull request: yes

# What does this PR do?

This PR specifies `framework="tf"` for `ImageToTextPipelineTests.test_small_model_tf`.

--------

The pipeline tests are run in 3 docker environment: only PT, only TF, both PT/TF installation.

In the last case (`PT/TF installed`), the loaded models in tests like `test_small_model_tf` are in fact **PT models**, as we didn't specify the framework when creating the pipelines.

This is particularly confusing during debugging - especially when a model will produce diff. outputs between PT/TF models.
In such case, the same test will fail either in TF-only env. or in PT/TF env.

(One example is PR #19565, where I got the expected values from PT/TF env., and it failed on CI for TF-only env.)

**TODO**:
- Same change to other TF pipeline tests
- Investigate why we get diff. outputs between PT/TF models for `ImageToTextPipelineTests.test_small_model_tf`.




Comments:
> Investigate why we get diff. outputs between PT/TF models for ImageToTextPipelineTests.test_small_model_tf.

Yup this is weird indeed, but glad we're catching such bugs !

_The documentation is not available anymore as the PR was closed or merged._

@Narsil Yes, that's why I don't want to change other places in this PR 😃 

I know what's wrong with PT v.s. TF 👀 !
Fix activations being all the same module
Author: sgugger
Pull request: yes

# What does this PR do?

Since #15616, all activations of the same type are the same instance of the same object. This seems harmless at first glance, but cause issues when hooks are automatically added to models, for instance in the big model inference. In this case all objects share the same hook (since they are all the same) leading to subtle bugs (kudos to @ArthurZucker and @younesbelkada for finding the root cause!)

This PR changes the `ACT2FN` dictionary to return new instances of the activation classes each time you provide a key.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add decorator to flaky accelerate test
Author: amyeroberts
Pull request: yes

# What does this PR do?

Adds `is_flaky` decorator to the `test_run_squad_no_trainer` test which occasionally fails on CI runs which are independent  to the changes in the PR e.g.: 

* https://app.circleci.com/pipelines/github/huggingface/transformers/49621/workflows/14c25312-58a5-4b0b-8b41-6c5bec668043/jobs/593213
* https://app.circleci.com/pipelines/gh/huggingface/transformers/49224/workflows/fbae76ab-9259-4695-bb06-475357172587/jobs/589262


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Great! Feel free to merge and open an issue to track it :)

I've opened an issue here: https://github.com/huggingface/transformers/issues/19733
Optimizer being initialized twice when using Nvidia APEX mixed precision backend
Author: JohnnyRacer
Pull request: no

### System Info

- `transformers` version: 4.21.1
- Platform: Linux-5.4.0-128-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.12.1+cu113 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False
- GPU : Nvidia A4000

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am trying to fine tune models on Jupyter notebooks. When training for the first time using APEX by calling `trainer.train()` it works fine. But if I start another run or try to train another model without restarting the kernel and calling the same function I get an error `RuntimeError: A given optimizer should only be passed through amp.initialize once`  . I thought it was because I had used the `adamw_apex_fused` setting for the optim in the`TrainingArguments`, but it happens regardless of whether or not I specify a parameter for the optim. I believe this is a possible bug due to `transformers` calling the `amp.initialize` function every time the trainer is started and does not check if `apex.amp` has been instantiated. An example of my training code is provided below, I would greatly appreciate any help. Thanks!

```python

training_args = TrainingArguments(
    f"gpt2-wikitext",
    evaluation_strategy = "epoch",
    learning_rate=2e-5,
    weight_decay=0.01,
    half_precision_backend="apex",
    tf32=True,
    fp16=True,
    auto_find_batch_size=True,
    optim='adamw_apex_fused'
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=lm_datasets["train"],
    eval_dataset=lm_datasets["validation"],
)

trainer.train()

```


### Expected behavior

Training should proceed normally without errors if `trainer.train()` is called multiple times in the same session.

Comments:
I can confirm this is done at every call to train, though I don't see an easy way of undoing it and we stopped actively maintaining the integration with Apex now that mixed precision is upstreamed in PyTorch.

Ah I see, so I should consider Apex support to be deprecated in `transformers` and move all future code to use native PyTorch MP backend?

Yes, you'll get better support this way!

That's great, thanks a lot for clearing things up!
[Doctest] - Fixing doctest `configuration_pegasus_x.py`
Author: mukesh663
Pull request: yes

# What does this PR do?
Fixes #19487 
Added (with random weights) in `configuration_pegasus_x.py`.
Added `configuration_pegasus_x.py` in `documentation_tests.txt`.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@ydshieh 


Comments:
Hi @mukesh663 There seems to be some CI issue. Could you push again with an empty commit?

You can do `git commit --allow-empty` then push :pray.

Hi, @mukesh663 I haven't merge your PR yet ...but you already closed the PR and delete the branch 😢 

Are you able to restore the branch ?

> Are you able to restore the branch ?

@ydshieh Sorry for the mishap. I have restored the branch.


Merged, thank you again!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19725). All of your documentation changes will be reflected on that endpoint.
Adding -> Configuration_flava.py
Author: AShreyam
Pull request: yes

# What does this PR do?

Add configuration_flava.py to utils/documentation_tests.txt for doctest.

Based on issue https://github.com/huggingface/transformers/issues/19487

@sgugger could you please take a look at it?
Thanks =)

Comments:
Hi @AShreyam 

This model config is done in #19724. Sorry if I forgot to follow this PR earlier. Going to close the PR though.
Thank you however.
[Doctest] Add configuration_big_bird.py
Author: lappemic
Pull request: yes

1. Change the import order of the model and configuration classes
2. Add `configuration_big_bird.py` to `utils/documentation_tests.txt` for doctest.

Documentation edit according to #19487

@sgugger could you have a look on this?

Comments:
This is already being fixed in https://github.com/huggingface/transformers/pull/19606

Hey @Xabilahu ok, i did not see this, sorry. I though do not understand fully, your pull request regards `configuration_bigbird_pegasus.py` and mine `configuration_big_bird.py`. How is it, that these are belong to the same?

I address both models in my PR. 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19619). All of your documentation changes will be reflected on that endpoint.

just saw it. Sorry, did not pay enough attention!



Hi @lappemic 

Still thank you. If you want to contribute, you can check [documentation_tests.txt](https://github.com/huggingface/transformers/blob/main/utils/documentation_tests.txt) on the main branch and see which ones are still missing :-)