noamaanMulla-03/transformers-issues huggingface dataset:
`is_ctc` needs to be updated to `self.type == "ctc".
Author: Narsil
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

Comments:
REALM
Author: aced125
Pull request: no
Labels: name: New model

name: Feature request

# 🌟 New model addition

REALM is from some of the authors of BERT (I like to think of it as the next BERT :) ) that have found a way to incorporate world knowledge (from Wikipedia) into the model.

They do this by having the concept of a retriever module that retrieves information from wikipedia articles.

<!-- Important information -->

## Open source status

Code not released at the moment but will probably be released by Google soon I'd imagine.

https://arxiv.org/abs/2002.08909


Comments:
Code is released for the preceding paper: Latent Retrieval for Weakly Supervised Open Domain Question Answering" https://www.aclweb.org/anthology/P19-1612/
https://github.com/google-research/language/tree/master/language/orqa

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I am also interested on it. Any news? Thanks

I'm also interested on this, are you planning to include this model in the library??

Finally the REALM code has been released here https://github.com/google-research/language/tree/master/language/realm.
I think that this issue should be re-opened. @aced125  are you able to re-open it?

I hope somebody can reopen this and make it happen

Any updates on this ? 

Any update? 


I think this could be implemented easily with RAG by making sure that we
both finetune doc encoder and question encoder of the retriever model. This
might be very useful for us.

On Wed, Feb 10, 2021, 23:42 Lysandre Debut <notifications@github.com> wrote:

> Reopened #3497 <https://github.com/huggingface/transformers/issues/3497>.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/3497#event-4314060510>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGULJ5VEDTMJD6F4ZXLS6JPJPANCNFSM4LVQ3UVQ>
> .
>


Any Update on this?

@OctoberChang  I did extend the RAG in a way that can be used to experiment REALM stuff. 

https://paperswithcode.com/paper/fine-tune-the-entire-rag-architecture

@shamanez , thanks for the pointer to your RAG project [(link)](https://github.com/huggingface/transformers/tree/master/examples/research_projects/rag-end2end-retriever)
Do you have any pre-trained (not fine-tuned on downstream tasks) models with ICT or Salient span masking [[link]](https://arxiv.org/pdf/2002.08909.pdf) that can be load from Huggingface Transformers?

I haven't extended into the ICT. Sorry.
@OctoberChang  I think to get REALM models loaded with the current HF retrieval framework, we might need some workarounds. This is mainly because RAG uses a generative model and REALM consists of extractive LM. 

@qqaatw's WIP PR (https://github.com/huggingface/transformers/pull/13292) adds REALM.
Question about fine-tuning with vision transformer (VIT)
Author: YHYeooooong
Pull request: no

Thanks for sharing great work!
I have a simple question about fine-tuning with VIT.
Is there any option for selecting trainable weight during fine-tuning phase? Or can I only train all weight?

Comments:
Hi,

You can easily disable parameters which you don't want to train with requires_grad=False:

```
from transformers import ViTForImageClassification

model = ViTForImageClassification.from_pretrained("google/vit-base-patch16-224-in21k")

for name, param in model.named_parameters():
   if name == "...":
      param.requires_grad = False
```

Thank you for your fast and kind response! I will try it!
Question about fine-tuning BeitForSemanticSegmentation model
Author: KombangkoeDias
Pull request: no

From the documentation, it says that the logits shape will be (batch_size, num_labels, height/4, width/4) 
I assume that the logits are the output masks of the model (since I'm doing the segmentation).
How do I convert this shape (height /4, width /4) to the original image's shape before being resized to (height, width)? 

-> # logits are of shape (batch_size, num_labels, height/4, width/4)
I realized that the input image is resized to the shape (height, width) by BeiTFeatureExtractor object
while the height and widthare those defined in the BeitFeatureExtractor's config and are constant values.
This means the output values' shapes are not the original image's shape but are rather resized-shape/4

Comments:
Hi,

You can take a look at this notebook: https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SegFormer/Fine_tune_SegFormer_on_custom_dataset.ipynb

It shows how to fine-tune SegFormer on a custom dataset (BeiT is equivalent), and it also outputs logits of shape (batch_size, num_labels, height/4, width/4). One interpolates these to the original size of the image. 

Note that we are in the process of determining generic outputs for semantic segmentation models, so it might be that in the future, the logits will automatically have the same size as the original `pixel_values`.



Thank you.
Add MAE
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR implements [MAE](https://github.com/facebookresearch/mae) (Masked Autoencoders are Scalable Vision Learners) by Facebook AI.

I've implemented it as a separate model in the library, as the model is quite specific (encoder only operates on visual patches, next a mask token is added and the decoder reconstructs pixel values based on encoded visual patches + mask tokens). The model without decoder on top is called `ViTMAEModel`, the model with decoder on top is called `ViTMAEForPreTraining` (inspired by `BertForPreTraining`). 

After pre-training, one can load the weights directly into a `ViTForImageClassification`.

* Models are on the hub: https://huggingface.co/models?other=vit_mae

* Demo notebook: https://colab.research.google.com/drive/1edgG0ne4VNQrc11uAS0wzogg5HblzK2s?usp=sharing

To do:

- [x] fix some tests. However, these fail due to the non-deterministic behaviour of the model (a random mask is generated in each forward pass). I can make the integration test deterministic by adding `torch.manual_seed(2)`, however adding this on top of the test file doesn't make it deterministic for other tests.

Comments:
Using the accelerate MLM example still results in CUDA out of memory
Author: tanyaroosta
Pull request: no

Hi,
I am using [this example](https://github.com/huggingface/transformers/blob/master/examples/pytorch/language-modeling/run_mlm_no_trainer.py) and instructions on [this discussion](https://discuss.huggingface.co/t/how-to-use-specified-gpus-with-accelerator-to-train-the-model/10967) to adapt the code from [run_summarization.py](https://github.com/huggingface/transformers/blob/master/examples/pytorch/summarization/run_summarization.py), and train (fine tune) a summarization BART model (specifically     model_name="sshleifer/distilbart-xsum-12-6" that I got from HF model repo).  I have one GPU and my batch size is 8.  My training data sample size is 15k. 
 However, as soon as the training starts, I get the following error:

`RuntimeError: CUDA out of memory. Tried to allocate 512.00 MiB (GPU 0; 14.76 GiB total capacity; 13.49 GiB already allocated; 419.75 MiB free; 13.54 GiB reserved in total by PyTorch)`

My full code can be found [here](https://github.com/tanyaroosta/test/blob/master/accl_trainer.py).  

Other relevant parameters:
cuda available" True
current device: 0
device count: 1
device name: Tesla T4
Allocated: 1.1 GB
accelerator device" cuda
***** Running training *****
  Num examples = 15909
  Num Epochs = 3
  Instantaneous batch size per device = 8
  Total train batch size (w. parallel, distributed & accumulation) = 8
  Gradient Accumulation steps = 1
  Total optimization steps = 5967

I would appreciate any help as I am at a loss of what else to try to get this training to go through.  I am using a SageMaker instance of ml.g4dn.16xlarge.

Thanks for your help.

Comments:
Hi,

Can you try out this: https://github.com/rentruewang/koila

Let me know if it resolves your issue.

koila didn't work for me. I basically had to use a p3.24 instance to get it to work with a batch size of 1.
added functionality to output the probabilities of the generated tokens #5164
Author: guyeyal
Pull request: yes

see https://github.com/huggingface/transformers/issues/5164

running:

    from transformers import AutoTokenizer, GPT2LMHeadModel, AutoModelWithLMHead
    model_name = 'facebook/bart-base'
    print("load tokenizer")
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    print("load model")
    model = AutoModelWithLMHead.from_pretrained(model_name, pad_token_id=tokenizer.eos_token_id if tokenizer.pad_token_id is None else tokenizer.pad_token_id)
    tokenizer_kwargs = {"add_prefix_space": True}
    s1 = "Hello! Looks like you’re enjoying the discussion, but you haven’t signed up for an account yet. When you create"
    # s2 = "wow, that's cool"
    tokens = tokenizer(s1, return_tensors='pt', **tokenizer_kwargs, )
    # input_ids = tokenizer.encode(str, return_tensors='pt',  **tokenizer_kwargs)
    outputs, probs = model.generate(input_ids=tokens['input_ids'], return_probs=True, num_beams=3, num_return_sequences=3, max_length=100)
    outputs = model.generate(input_ids=tokens['input_ids'], num_beams=3,
                                    num_return_sequences=3, max_length=100)

Comments:
Hi, is there any reason why this function not merged into the generate function? Hope to use this functionality to incorporate with reinforcement learning in my work.

Hey @ngoquanghuy99,

Sorry to answer only now. It is already possible to output probabilities as shown here: https://discuss.huggingface.co/t/generation-probabilities-how-to-compute-probabilities-of-output-scores-for-gpt2/3175/15?u=patrickvonplaten
Fix incorrect max_seq_length bug in pytorch training scripts
Author: novice03
Pull request: yes

# What does this PR do?

Fixes #15181 

Input sequences are not correctly truncated to the correct length in the `run_mlm.py` script. Even if a model can support large sequence lengths (such as 1024 or 2048), the sequences are truncated to the tokenizer's default `model_max_length`. This bug is described [here](https://github.com/huggingface/transformers/issues/15181).

This pull request fixes this bug by comparing the `max_seq_length` passed by the user to the maximum sequence length supported by the model (`config.max_position_embeddings`), and sets `max_seq_length = min(data_args.max_seq_length, config.max_position_embeddings)`

If this is the correct fix, I can make the same change in other pytorch training scripts such as [run_plm.py](https://github.com/huggingface/transformers/blob/9a2dabae7002258e41419491c73dd43ad61b5de7/examples/pytorch/language-modeling/run_plm.py#L363), [run_qa.py](https://github.com/huggingface/transformers/blob/9a2dabae7002258e41419491c73dd43ad61b5de7/examples/pytorch/question-answering/run_qa.py#L333), etc.

## Who can review?

@NielsRogge @LysandreJik @sgugger 

Comments:
As said on the linked issue, this should not be changed.
Error when running TFT5ForConditionalGeneration with tensorflow-cpu==2.8.0-rc0
Author: tdomagal
Pull request: no

### Environment info
- `transformers` version: 4.15.0
- Platform: Linux-5.4.0-88-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.8.0-rc0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
- T5: @patrickvonplaten
or
- TensorFlow: @Rocketknight1

## Information

Model I am using T5:


The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)



## To reproduce

Steps to reproduce the behaviour:

1. upgrade tensorflow to 2.8.0

> pip install -U tensorflow-cpu==2.8.0rc0

2. follow example, like in https://github.com/snapthat/TF-T5-text-to-text/blob/master/snapthatT5/notebooks/TF-T5-%20Training.ipynb
3. error raised during fit call:
```
        ValueError: Found unexpected losses or metrics that do not correspond to any Model output: dict_keys(['loss']).
 Valid mode output names: ['output_1']. Received struct is: {'loss': <function dummy_loss at 0x7f7a2d430e50>}.
```

## Expected behavior

No error


Comments:
I'm seeing the same issue on official example: examples/tensorflow/question-answering

Can you confirm if this issue still occurs when using TF 2.6 or 2.7?

In case of 2.7.0 issue does not reproduce.

The following steps allow of easy reproduce using transformers/examples/tensorflow/question-answering:

```
python -m virtualenv -p python3 venv
source venv/bin/activate
pip install tensorflow==2.8.0-rc0

git clone https://github.com/huggingface/transformers
cd transformers
pip install .

cd examples/tensorflow/question-answering
pip install -r requirements.txt

pip install torch

python run_qa.py --model_name_or_path distilbert-base-cased --output_dir output --dataset_name squad --do_train --do_eval
```



Hmm, okay, this seems like a compatibility issue with TF 2.8 that we'll have to resolve soon. I'd guess it's likely caused by us overriding `train_step` in our code, but TF 2.8 changing the default `train_step` to handle the new `Model.compute_losses` and `Model.compute_metrics`.

Will discuss with the team and see if we can figure out a solution before the final 2.8 release. Thank you for the bug report! 

@tdomagal @atom00 We have identified the issue - the Keras method `compute_loss` which was newly added in 2.8 is clobbering the `compute_loss` method that all HuggingFace models already had. We're going to rename our method and see if that resolves things. Thank you again for this report - it alerted us to an important issue before 2.8 went live!
Build dev documentation
Author: LysandreJik
Pull request: yes


Comments:
Thank you for your PR. The documentation will now be removed from the staging environment - feel free to reopen this PR to recreate it.

Thank you for your PR! The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15110).\n\nAll of your documentation changes will be reflected on that endpoint.

Thank you for your PR. The documentation will now be removed from the staging environment - feel free to reopen this PR to recreate it.

Thank you for your PR! The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15110). All of your documentation changes will be reflected on that endpoint.

Thank you for your PR. The documentation will now be removed from the staging environment - feel free to reopen this PR to recreate it.

Thank you for your PR! The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_$PR_NUMBER). All of your documentation changes will be reflected on that endpoint.

Thank you for your PR. The documentation will now be removed from the staging environment - feel free to reopen this PR to recreate it.

Thank you for your PR! The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_$PR_NUMBER). All of your documentation changes will be reflected on that endpoint.

Great job merging this PR! the documentation will now be removed from the staging environment.
[ViTMAE] Various fixes
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR:
- adds a link to a demo notebook for Facebook's MAE.
- fixes the links to the docs of various Vision Transformer-based models in ViT's doc page.
- fixes the code examples for MAE.
- adds MAE to the `AutoFeatureExtractor` API.

Comments:
Great job merging this PR! the documentation will now be removed from the staging environment.
[Speech Event] Fix speech event readme
Author: patil-suraj
Pull request: yes

# What does this PR do?



Comments:
Thank you for your PR! The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_$PR_NUMBER). All of your documentation changes will be reflected on that endpoint.

Thank you!

Great job merging this PR! the documentation will now be removed from the staging environment.
Fix PR number
Author: LysandreJik
Pull request: yes

Updates the reference to the environment variable to respect Github Actions' format.

Comments:
Thank you for your PR! The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_${PR_NUMBER}). All of your documentation changes will be reflected on that endpoint.

Thank you for your PR! The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15231). All of your documentation changes will be reflected on that endpoint.

Thank you for your PR. The documentation will now be removed from the staging environment - feel free to reopen this PR to recreate it.

Great job merging this PR! the documentation will now be removed from the staging environment.
Add ViLT
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds [ViLT](https://arxiv.org/abs/2102.03334) (Vision and Language Transformer).

It's a very nice, minimal multi-modal model, as it only adds a text embedding layer to an existing ViT.

I've defined the following head models:

* `ViltForMaskedLM`
* `ViltForVisualQuestionAnswering`
* `ViltForNaturalLanguageVisualReasoning`
* `ViltForImageRetrievalTextRetrieval` (CLIP-like model).

To do:

- [x] add `ViltForNaturalLanguageVisualReasoning` to the tests. However, I do have a question here: it's the only model that requires `config.modality_type_vocab_size = 3` instead of 2. How can I handle this exception in the tests? I could do it like this:

```
for model_class in self.all_model_classes:
    config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()
    config.return_dict = True

    if model_class.__name__ == "ViltForNaturalLanguageVisualReasoning":
       config.modality_type_vocab_size = 3
```
But that's not ideal as it would require overwrite each individual test.

Update: fixed by create a separate `ModelTester` for this particular model, that overrides the `get_config`.

Comments:
I'd like the PR to be green (or mostly green) before reviewing.

@sgugger should be mostly green now.

Note: with the new build dev job merged, you can preview the doc [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_14895/en/model_doc/vilt) :-) 

Great job merging this PR! the documentation will now be removed from the staging environment.
Update Trainer code example
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR updates the code example used in the Trainer docs. Previously, it shows how to overwrite the Trainer to do multi-label classification. However, this is not required anymore, as users can now pass the `problem_type` argument to the model's configuration (to use the appropriate loss function).

Instead, I show how to overwrite the Trainer to use a weighted loss, useful when you have an imbalanced dataset.


Comments:
Great job merging this PR! the documentation will now be removed from the staging environment.
Add support for BERT SequenceClassification conversion to ONNX
Author: naimenz
Pull request: no

# 🚀 Feature request

Currently, the `SequenceClassification` pipeline is not supported for converting BERT-based models to ONNX. Support for this pipeline would be great.

## Motivation

As part of a recent internship, I needed to deploy a `SequenceClassification` `SciBERT` model by first converting it to ONNX and quantizing it. I wrote a script that did this, and so thought I should contribute this to the repo.

## Your contribution

I'd be happy to make a PR, but I haven't contributed before so I thought I'd make an issue first to check that there were no problems I hadn't seen before diving in.

For example:
* Is this something that is already being worked on? (I couldn't see an open PR or issue related to it, but maybe there's other work I don't know about)
* Are there other factors that would complicate this work? (It was pretty simple to quantize the individual models I was working with, but I can imagine there being more difficulties in implementing it generally)

Comments:
I'm pretty sure `xxxForSequenceClassification` models are supported. You can read the updated docs [here](https://huggingface.co/docs/transformers/master/en/serialization).

For instance, you can convert one as follows:

```
python -m transformers.onnx --model=distilbert-base-uncased-finetuned-sst-2-english \
                            --feature=sequence-classification onnx/
```

Ah you're right, I must have somehow had an older version, really sorry about that! Next time I'll try to make a first issue that is actually an issue.