noamaanMulla-03/transformers-issues huggingface dataset:
Add 'with torch.no_grad()' to DeBERTa integration test forward pass
Author: h-holm
Pull request: yes

# What does this PR do?

As proposed in #14642, this encapsulates the forward pass in the DeBERTa integration test with "with torch.no_grad():". This way, no unnecessary gradients are computed during inference.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?
@NielsRogge 

Comments:
Add 'with torch.no_grad()' to BERT integration test forward pass
Author: h-holm
Pull request: yes

# What does this PR do?

As proposed in #14642, this encapsulates the forward pass in the BERT integration test with "with torch.no_grad():". This way, no unnecessary gradients are computed during inference.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?
@NielsRogge 

Comments:
Roberta Classification Head
Author: arijitthegame
Pull request: no

Why does Roberta classification Head uses 2 linear layers? https://github.com/huggingface/transformers/blob/84ea427f460ffc8d2ddc08a341ccda076c24fc1f/src/transformers/models/roberta/modeling_roberta.py#L1443 

The BERT classification head uses only one and it makes sense since all the work is done by the Transformer layers. 

Comments:
Hi! We try to stick as much as possible to the original implementation. See fairseq's original implementation here: https://github.com/pytorch/fairseq/blob/a54021305d6b3c4c5959ac9395135f63202db8f1/fairseq/models/roberta/model.py#L394-L429

Thank you so much.
FlaxVisionEncoderDecoderModel has no attribute 'from_encoder_decoder_pretrained' ?
Author: PrithivirajDamodaran
Pull request: no

Transformers Version: 4.14.1

`FlaxVisionEncoderDecoderModel.from_encoder_decoder_pretrained('google/vit-base-patch32-224-in21k', 'gpt2')`

throws the following error

`AttributeError: type object 'FlaxVisionEncoderDecoderModel' has no attribute 'from_encoder_decoder_pretrained'`

But the documentation below suggests its available 

https://huggingface.co/docs/transformers/model_doc/visionencoderdecoder

what am I missing?

Comments:
I think this is because `flax` is not installed, for flax models we should install `jax`, `jaxlib` and `flax`. The import returns a dummy `FlaxVisionEncoderDecoderModel` when flax is not installed which has no methods.

You could see in this colab that it works https://colab.research.google.com/drive/1g6oU-AYFpzVNWgn1qT5NKlEqor3tLCeI?usp=sharing

Thanks @patil-suraj
Add a main_input_name attribute to all models
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds a new attribute to all models called `main_input_name` which allows us to know whether the model expects `input_ids`, `pixel_values`, `input_values` or `input_features` as a first argument. A test is also added to check the correct value is set for all models, by inspecting the signature.

We could also add some magic property to do what's tested, but I feel an explicit attribute is more customizable for future use cases.

Comments:
how to ensemble different checkpoints?
Author: shawnkx
Pull request: no
Labels: name: wontfix

I want to ensemble different checkpoints trained from the same parameter configuration but different seeds. Could you tell me how to ensemble these checkpoints?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@shawnkx Hi! Have you found a solution？

@all, any updates on this?
T5ForConditionalGeneration lm_head not initialized from pretrained checkpoint
Author: ZhaofengWu
Pull request: no

It seems that pretrained T5 checkpoints such as `"t5-small"`, `"t5-base"`, etc., do not contain the `lm_head` parameter.
https://github.com/huggingface/transformers/blob/84ea427f460ffc8d2ddc08a341ccda076c24fc1f/src/transformers/models/t5/modeling_t5.py#L1434-L1438
@patrickvonplaten explained in #3553 that these three parameters are shared, but, while I agree that the encoder and decoder embeddings are shared, I don't think `lm_head` is also shared with them. This is because (1) I couldn't find any such sharing logic in `modeling_t5.py`, and (2) these weights are separate in the original TF checkpoint.

Comments:
Hey @ZhaofengWu, 

For all encoder-decoder models, such as T5, Bart, ..., the `lm_head` of the decoder is shared between the decoder input word embedding if `tie_word_embeddings` is set to `True` in the model's config. You can check the config for `t5-small` [here](https://huggingface.co/t5-small/blob/main/config.json) *e.g.*
If the config doesn't have this parameter set, then one should look at the default value in `configuration_utils.py` here: https://github.com/huggingface/transformers/blob/c1125dc2ba9f3c383bf860ac9fcd67268385ad8d/src/transformers/configuration_utils.py#L256 where it is set to ``True``.

In TF the weighs are also shared 

Ahh ok, thanks for pointing me to where the tying takes place. Also I didn't realize that T5 v1.1 and T5 have different behaviors here, and had this question since I saw separate parameters in the TF checkpoint for v1.1.
How to use Roberta as the Encoder and a randomly initialized TransformerDecoder as the Decoder?
Author: Captainr22
Pull request: no

Hello！
I want to use Roberta as an Encoder and a randomly initialized TransformerDecoder as a Decoder. I know EncoderDecoderModel can make any pretrained autoencoding model as the encoder and any pretrained autoregressive model as the decoder. But I just want to use 4 or 6 TransformerDecoder layers as my decoder. 
Maybe I should use torch.nn.TransformerDecoder to build my model. But is there any easier way or example when I use Huggingface Transformers?
Thank you!
@patrickvonplaten @NielsRogge

Comments:
Hey @Captainr22, 

I would recommend doing the following:

- 1) Create a randomly intialized `RobertaModel` as your transformer decoder:
```python
from transformers import RobertaModel, RobertaConfig

decoder = RobertaModel(RobertaConfig(num_layers=4))
decoder.save_pretrained("./temp")
```

- 2) Now load an encoder / decoder model as follows:

```python
from transformers import EncoderDecoderModel

model = EncoderDecoderModel.from_encoder_decoder_pretrained("roberta-base", "./temp")
```



Thank you for your recommendation！
[logging] implement warning_advice / TRANSFORMERS_NO_ADVISORY_WARNINGS
Author: stas00
Pull request: yes

This PR implements the idea of advisory logger warnings as discussed at https://github.com/huggingface/transformers/issues/14455

The main ideas is to enable users to turn some logger warnings off when they re-run the same program again and again and either don't care or can't do something about those warnings. Additionally, this is essential for multi-node setups where the same warning gets replicated for each process and with 512 processes it's a lot of identical warnings.

This PR:

- adds a new logger method `logger.warning_advice` which works just like `logger.warning` but which is a no-op if the env var `TRANSFORMERS_NO_ADVISORY_WARNINGS=1` is set
- adds tests
- updates docs
- replaces one warning for now - need to replace more.

TODO:

- decide which warnings should always remain warnings and which can be switched to advisory ones, this can also be a gradual process. We can also invite input from the community.

- Once completed I'd also propose to replicate this to `datasets` as most of the time repeat uses and the warnings are just as non-informative and ignored there as in `transformers`.

Fixes: https://github.com/huggingface/transformers/issues/14455

@sgugger, @LysandreJik, @patrickvonplaten, @patil-suraj

Comments:
Looks good to me! I'm wondering whether we actually need a new logger function `logger.warning_advice` or whether it might make sense to simply overwrite `logger.warning` since 100% backward compatibility is kept 

> Looks good to me! I'm wondering whether we actually need a new logger function `logger.warning_advice` or whether it might make sense to simply overwrite `logger.warning` since 100% backward compatibility is kept

You're thinking about having a mechanism of turning all warnings off, aren't you?

If so this is already available via logger's set verbosity.

If we are going to change all warnings to advisory warnings then this feature is pointless. 

My thinking was that some warnings are a must whereas others are either border-line `info`-level or they are not close to `error` level.

Perhaps we need to have a closer look at the types of warnings we have right now. If there no clear sub-division of which warnings are essential and which aren't, then this feature won't help.

I'm personally ok with the `warning_advice` which is supposed to be different to `warning` - for me it's good to merge

Ok for me!
Make the onnx submodule init lazy
Author: sgugger
Pull request: yes

# What does this PR do?

The `onnx` submodule does not use a lazy init like the other ones. This results in importing a given model/tokenzier/config, which imports OnnxConfig, initializing the onnx submodule completely, and in turn importing `PreTrainedModel` and `TFPreTrainedModel` (so PyTorch and TensorFlow).

This PR solves this issue by making the init lazy like all the others.

Comments:
trainer.create_model_card should be run by process 0 only in distributed training
Author: dunalduck0
Pull request: no

### Who can help
@gchhablani, @sgugger

## Information

Trainer.create_model_card() does not check which process is running. In distributed training, all processes would try to write into $output_dir/README.md and sometimes cause crashes.

The problem arises when using run_clm.py in distributed training.

## To reproduce

Steps to reproduce the behavior:

1. Install transformers from local source by pip install -e .
2. Make two small changes to create_model_card() in trainer.py, in order to highlight the issue:
    (1) added a logging into trainer.py right before writing README.md
     (2) appended the output filename with process index.

(Sorry about incorrect indents. I can't make spaces work here)
def create_model_card(self,...):
        training_summary = TrainingSummary.from_trainer( self, language=language,...)
         model_card = training_summary.to_model_card()
         **logger.warning(f"Process {self.args.process_index} writes to README.md!")**
         with open(os.path.join(self.args.output_dir, "README.md" **+ str(self.args.process_index))**   ), "w") as f:
         f.write(model_card)

3. Try run_clm.py with any training task with 2 GPUs, push_to_hub = false, and the modified create_model_card()
4. The end of the script will call trainer.create_model_card(**kwards). You should see from logging both processes tried to create a model card file. The output directory should have two README.md files, one from process 0 and the other from process 1. Without the filename change, the two processes would have tried to write to the same README.md and cause racing in distributed training. I have one of such crashes when training in GPU clusters.

My logging:
```
[INFO|modelcard.py:456] 2021-12-19 19:59:04,324 >> Dropping the following result as it does not have all the necessary fields:
{'task': {'name': 'Causal Language Modeling', 'type': 'text-generation'}}
[WARNING|trainer.py:2643] 2021-12-19 19:59:04,325 >> Process 0 writes to README.md!
[WARNING|trainer.py:2643] 2021-12-19 19:59:04,334 >> Process 1 writes to README.md!
[2021-12-19 19:59:07,398] [INFO] [launch.py:160:main] Process 540006 exits successfully.
[2021-12-19 19:59:08,400] [INFO] [launch.py:160:main] Process 540007 exits successfully.
```

Two README.md in my output directory
```
(gptj) REDMOND.meiyang@GCRSANDBOX354:/tmp/model_output$ ll -t /tmp/model_output
total 23753748
drwxr-xr-x  7 xxxxxxx       4096 Dec 19 19:59 ./
-rw-r--r--  1 xxxxxxx        1108 Dec 19 19:59 README.md1
-rw-r--r--  1 xxxxxxx        1108 Dec 19 19:59 README.md0
```

## Expected behavior

trainer.create_model_card() should allow only process 0 to write so that only one README.md is written


Comments:
Thanks for flagging! Adding some logic for that in the PR mentioned above.
Add custom `stopping_criteria` and  `logits_processor` to `generate`
Author: lvwerra
Pull request: yes

# What does this PR do?

This PR are continues the work and discussions from #12219 with a fresh start.

It integrates the custom `stopping_criteria` and `logits_processor` with the following logic: 
- the default `stopping_criteria`/`logits_processor` are created from the arguments and model's config 
- if additional, custom `stopping_criteria` and `logits_processor` are passed to `generate`, they are compared with the default list
   - if there is an overlap between the two lists (e.g. a `MaxLengthCriteria` in both lists) an error is thrown
   - if there is no overlap the two lists are merged 

Fixes #12118

Comments:
@Narsil thanks for your feedback and catching that copy-paste error! I added something to the docstring and fixed the error.

@patrickvonplaten any comments?
Convert docstrings of modeling files
Author: sgugger
Pull request: yes

# What does this PR do?

This is going to be a huge PR (sorry) but all modeling files have docstrings that are connected to the model outputs and the code samples in file_utils, so the conversion from rst to Markdown has to happen for all of those at the same time.

The doc-styler is commented out for now, as it can't deal with example written in Markdown. I will work on re-enabling it as soon as we have finished converting all the documentation.

Separately, this PR deals with an issue of the Return block in some dosctrings being indented at 4 spaces instead of the indentation inside the docstring, which resulted in the Example block as being part of the return block.

Comments:
Saw with @sgugger and the newline before `"""` isn't important and isn't liked by black.

As seen with Sylvain and Patrick, merging this PR now to prevent any conflicts - let's take a look at `master` once deployed to ensure everything works correctly.

From what I'm seeing, everything looks correct, including code samples! Thanks for your work, @sgugger!
[doc porting] several docs
Author: stas00
Pull request: yes

Ported to mdx:
```
docs/source/debugging.rst
docs/source/testing.rst
docs/source/main_classes/deepspeed.rst
```

fixed a few small code block breakages in the first one.

@sgugger 

Comments:
Amazing that you noticed all those issues, @sgugger - I missed all of those.

It looks like we need to be careful with suggestions that include "```", it looks like github is buggy there, ate the "````" you added since it's also part of suggestion - had to switch to 4 backticks suggestion.
Mass conversion of documentation from rst to Markdown
Author: sgugger
Pull request: yes

# What does this PR do?

This PR treats a lot of docstrings and converts them from rst to Markdown. All of the remaining one.

It also includes a new repo consistency check that will detect someone does not add back some rst docstrings behind my back. 😈 

Comments:
As seen with @LysandreJik, merging and inspecting the result in the doc. Failure is unrelated.
[ONNX] Add symbolic function for XSoftmax op for exporting to ONNX.
Author: fatcat-z
Pull request: yes

# What does this PR do?

For such customized operator, PyTorch ONNX exporter will try to call its symbolic() function for exporting to ONNX file. Add the implementation here so that it could be exported successfully.


Comments:
Might be of interest to @BigBird01 

> Might be of interest to @BigBird01

This is cool! Thanks!

@BigBird01 
The latest CI failure seems like is not relative to my changes. Could you please take a review and see if we can merge this change?

Thanks.

> Seems good to me, thanks for your great contribution!
> 
> Quick question: do you plan to add the [ONNX DeBERTa export support](https://huggingface.co/transformers/serialization.html#exporting-transformers-models) now that you have added support for the problematic operator?

No, no plan for this yet.

@fatcat-z Hi, I have a question about the way symbolic conversion is implemented. Is there any reason why you are using `uint8` type for some values?
It seems to me to work with signed integers, and uint8 is not supported by TensorRT (it only supports signed integers).
Error while reproducing example for PerceiverForMultimodalAutoencoding
Author: mirth
Pull request: no

## Environment info
- `transformers` version: 4.14.1
- Platform: Linux-5.8.0-63-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.6
- PyTorch version (GPU?): 1.9.0+cu111 (True)
- Tensorflow version (GPU?): 2.5.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
@NielsRogge 

Trying to reproduce  [example](https://github.com/huggingface/transformers/blob/e51c7b5872785a74a03c011732173757d7c216c4/src/transformers/models/perceiver/modeling_perceiver.py#L1888) for `PerceiverForMultimodalAutoencoding` and getting:

```
>>> from transformers import PerceiverForMultimodalAutoencoding
2021-12-21 23:44:38.796858: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
>>> import torch
>>> images = torch.randn((1, 16, 3, 224, 224))
>>> audio = torch.randn((1, 30720, 1))
>>> inputs = dict(image=images, audio=audio, label=torch.zeros((images.shape[0], 700)))
>>> model = PerceiverForMultimodalAutoencoding.from_pretrained('deepmind/multimodal-perceiver')
>>> outputs = model(inputs=inputs)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tolik/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/tolik/anaconda3/lib/python3.7/site-packages/transformers/models/perceiver/modeling_perceiver.py", line 1912, in forward
    return_dict=return_dict,
  File "/home/tolik/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/tolik/anaconda3/lib/python3.7/site-packages/transformers/models/perceiver/modeling_perceiver.py", line 909, in forward
    inputs, modality_sizes, inputs_without_pos, subsampled_points=subsampled_output_points
  File "/home/tolik/anaconda3/lib/python3.7/site-packages/transformers/models/perceiver/modeling_perceiver.py", line 2429, in decoder_query
    [embed(modality, decoder_queries[modality]) for modality in sorted(self.modalities.keys())], dim=1
  File "/home/tolik/anaconda3/lib/python3.7/site-packages/transformers/models/perceiver/modeling_perceiver.py", line 2429, in <listcomp>
    [embed(modality, decoder_queries[modality]) for modality in sorted(self.modalities.keys())], dim=1
  File "/home/tolik/anaconda3/lib/python3.7/site-packages/transformers/models/perceiver/modeling_perceiver.py", line 2424, in embed
    pos = torch.broadcast_to(pos, [x.shape[0], x.shape[1], self.num_query_channels - x.shape[2]])
RuntimeError: The expanded size of the tensor (833) must match the existing size (831) at non-singleton dimension 2.  Target sizes: [1, 704, 833].  Tensor sizes: [1, 831]
```



Comments:
Fix pytorch image classification example
Author: mariosasko
Pull request: yes

Updates the `examples/pytorch/image-classification/run_image_classification.py` script to use the new Image feature to pass the failing [CI test](https://app.circleci.com/pipelines/github/huggingface/transformers/31642/workflows/b1aa40eb-2edc-4b54-963b-f11058072a7e/jobs/327951/artifacts) (I've also updated the `hf-internal-testing/cats_vs_dogs_sample` dataset on the Hub for that).

Comments:
Keras callback for prompting metrics
Author: merveenoyan
Pull request: yes

# What does this PR do?

It's a Keras callback that will print metrics at the end of every epoch. 
It's tested on summarization example in the examples.


## Who can review?

@Rocketknight1 @LysandreJik 


Comments:
Looks good so far! Here's what I'm thinking:

1. We should probably use the Callback's `self.model` instead of expecting a global `model` object - that might work in notebooks, but it's not great otherwise.
2. Looking at it, maybe you were right that we're asking the user to do too much work, and we should make predictions from the model in this callback too. It's definitely trickier, though, because Keras doesn't like concatenating variable-length predictions together. Maybe our code can predict on batches, break the batches into lists and then make big lists of predictions and labels?

Migrated to here: https://github.com/huggingface/transformers/pull/14867