noamaanMulla-03/transformers-issues huggingface dataset:
Add push_to_hub to no_trainer examples
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds the `--push_to_hub` flag to the `no_trainer` examples in the Transformers repository. For now, it only deals with `run_glue_no_trainer.py` but once everyone has commented, the final version will be duplicated on the other examples.

Comments:
this is great!

Also, since `blocking` argument to `push_to_hub` was only introduced `v0.0.17` maybe we should add this in the requirements.

@patil-suraj The requirements `huggingface_hub >= 0.0.17` is in the install requirements for Transformers now.
[AutoTokenizer] Allow creation of tokenizers by tokenizer type
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

This PR enables the `Case #4` as discussed here: https://github.com/huggingface/transformers/pull/13623#issuecomment-923112500


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
New Wav2Vec2 padding has slightly backward breaking changes
Author: patrickvonplaten
Pull request: no

The PR: https://github.com/huggingface/transformers/pull/13650 introduced some quite tricky backwards breaking changes that we should try to fix. 

The problem is the following: A user might directly use `feature_extractor.pad(...)` instead of `feature_extractor(...)` to just pad already preprocessed inputs in, *e.g.* a data collator.

The following code correctly returned `torch.float32` before merging the PR while the new PR returns `torch.float64` which is slighly breaking and can lead to errors in current fine-tuning Wav2Vec2 scripts:

```python
from transformers import Wav2Vec2FeatureExtractor
import numpy as np

extractor = Wav2Vec2FeatureExtractor.from_pretrained("facebook/wav2vec2-base-960h")

rand_input = np.ones((100,), dtype=np.float64)
out = extractor.pad([{"input_values": rand_input}], return_tensors="pt")

print(out.dtype) # <- this should be `torch.float32`
```

Here is a colab showing how the "old" version works correctly: https://colab.research.google.com/drive/10TlRWvwKx34UORmYdCFAyMWKUU3OtPRf?usp=sharing

Here is a colab showing how the "new" version works incorrectly: 
https://colab.research.google.com/drive/1cXGuG4Rnypmivdm-vdE-61BA1f4hC4e8?usp=sharing

Comments:
@anton-l - could you maybe look into it? :-) It's quite a tricky backwards compatible bug and we should have had tests to catch this problem. Would be great if you could try to open a PR to fix it :-)

Good catch! This is due to how pytorch converts float numpy arrays vs python lists:

* torch.float32 for python lists by default: `torch.tensor([1.2, 2.3]).dtype  #  torch.float32`
* `np.array([1.2, 2.3]).dtype  # np.float64` 
* source dtype for numpy arrays: `torch.tensor(np.array([1.2, 2.3]).dtype  #  torch.float64`
[Wav2Vec2FeatureExtractor] Fix `extractor.pad()` dtype backwards compatibility
Author: anton-l
Pull request: yes

Resolves #13689

This fixes an issue introduced by #13650 with speech feature extractors' tensors being returned as torch.float64 when `.pad()` is called directly:

```python
from transformers import Wav2Vec2FeatureExtractor
import numpy as np

extractor = Wav2Vec2FeatureExtractor.from_pretrained("facebook/wav2vec2-base-960h")

rand_input = np.ones((100,), dtype=np.float64)
out = extractor.pad([{"input_values": rand_input}], return_tensors="pt")

print(out.dtype) # <- this should be `torch.float32`
```

This is due to how pytorch converts float numpy arrays (new padding logic) vs python lists (old padding logic):

* uses torch.float32 for python lists by default: `torch.tensor([1.2, 2.3]).dtype  #  torch.float32`
* `np.array([1.2, 2.3]).dtype  # np.float64` 
* uses source dtype for numpy arrays: `torch.tensor(np.array([1.2, 2.3])).dtype  #  torch.float64`

Comments:
Limit number of checkpoint on `examples/pytorch/summarization/run_summarization.py` same as `save_total_limit` on `Trainer`
Author: frederick0291
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

Is it possible to limit the amount of checkpoints created in `examples/pytorch/summarization/run_summarization.py`. I finetuned a model and it keeps on creating a new checkpoint every 5000 steps. I know this is already implemented in transformers `Trainer` as `save_total_limit`

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

I finetuned my first model using Pegasus and the `run_summarization.py` script provided in the docs. I noticed that colab was always running out of memory so I had to manually delete checkpoints after every 1500 steps. This would make finetuning Pegasus models using `run_summarization.py` less tedious. 

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->

I could implement the `save_total_limit` param in `run_summarization.py` if this is not yet implemented. Just need to look on how to do it.


Comments:
Nevermind I think I figured it out. Looks like `run_summarization.py` inherits everything from `Trainer` so adding `--save_total_limit 2`  to the passed args limits the number of checkpoint savefiles.
Joint Sequence and Token Classification
Author: will-rice
Pull request: no

# 🚀 Feature request
A common NLU task for voice assistants and chatbots is intent classification and slot filling. This can be modeled as a joint task via a model like [BERT](https://arxiv.org/abs/1902.10909). The implementation is essentially a token classification head along with a sequence classification head attached to a pre-trained encoder. 

## Motivation

I think this could be a good addition because it would allow the upload of joint slot/intent models and allow a common NLU task to be added to pipelines. Furthermore, this would enable audio encoder models like Wav2Vec2 to be used for End-to-End SLU tasks like [this](https://arxiv.org/abs/1904.03670). 


## Your contribution

I would be open to contributing the implementation to BERT and uploading pre-trained NLU/SLU models on mainstream benchmarks like [snips](https://arxiv.org/abs/1805.10190v3) if other members of the community would benefit.


Comments:
Hello @will-rice, we're currently working on a feature to enable loading sharing and loading custom architectures like the one you mention cc @sgugger.

We'll let you know as soon as it's ready for testing so that you may take a look and let us know if it's helpful.
[Benchmark]
Author: Dennis0225
Pull request: no

# 🖥 Benchmarking `transformers`

## Benchmark

Which part of `transformers` did you benchmark?

## Set-up

What did you run your benchmarks on? Please include details, such as: CPU, GPU? If using multiple GPUs, which parallelization did you use?

## Results

Put your results here!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Patch training arguments issue
Author: LysandreJik
Pull request: yes

Fix an issue with the hub token. (v4.10 patch)

Comments:
LGTM! Modifying a dict as you iterate over it with `.keys()` is usually bad I think, since `keys()` returns a non-constant view that will change if the dictionary changes. In the case where you only update values and never add or remove any keys I believe it's okay, though.
[GPT-J] Use the `float16` checkpoints in integration tests
Author: anton-l
Pull request: yes

This PR switches GPTJ checkpoints in the integration tests to fp16 to test if they're able to run on our daily CI.
At the moment, fp32 checkpoints are timing out either during model downloads or initialization:
```
600.01s call     tests/test_modeling_gptj.py::GPTJModelTest::test_batch_generation
600.00s call     tests/test_modeling_gptj.py::GPTJModelLanguageGenerationTest::test_lm_generate_gptj
600.00s call     tests/test_modeling_gptj.py::GPTJModelLanguageGenerationTest::test_gptj_sample_max_time
600.00s call     tests/test_modeling_gptj.py::GPTJModelTest::test_model_from_pretrained
600.00s call     tests/test_modeling_gptj.py::GPTJModelLanguageGenerationTest::test_gptj_sample
```

Note that this doesn't guarantee reproducibility of the old tests (some tokens may be different), but it could help with **caching the models** on the runner to avoid timeouts.

:warning: The tests should be revisited once more, once https://github.com/huggingface/transformers/pull/13466 is merged

Comments:
`test_gptj_sample()` and `test_gptj_sample_max_time()` were disabled due to GPU OOM during more than one call to `.generate()`
FastFormers into transformers
Author: ykim362
Pull request: no
Labels: name: wontfix

name: New model

# 🌟 New model addition

## Model description

We just open-sourced [FastFormers](https://arxiv.org/abs/2010.13382) which are our SustaiNLP 2020 systems (FastFormers: Highly Efficient Transformer Models for Natural Language Understanding [paper](https://arxiv.org/abs/2010.13382)).

Currently, we are hosting this on our repository, but would like to merge it back to the transformers repository as an example.
our repo - https://github.com/microsoft/fastformers

For the purpose of the shared task, this is purely implemented with SuperGLUE data set.
So, it's dependent on Alex Wang(@W4ngatang)'s SuperGLUE data processing pipeline.
Also, many parts of the implementations are based on Alex'.
(https://github.com/W4ngatang/transformers/tree/superglue)

What would be the best way to merge this back?

## Open source status

* [x] the model implementation is available: https://github.com/microsoft/fastformers/blob/main/examples/fastformers/run_superglue.py
* [x] the model weights are available: demo systems are uploaded. https://github.com/microsoft/fastformers/releases/tag/v0.1-model
* [x] who are the authors: @ykim362 


Comments:
Hi! This is great, thanks for offering to contribute it! From what I understand, `FastFormers` contains several scripts that can be applied to `transformers` models out of the box, that is, training, distillation, pruning, using quantization alongside the onnx runtime and fp16 optimizations. 

Is that correct? If that is so, the easiest way would be to add the corresponding scripts to the `examples/` directory, probably under `examples/fastformers`. If there are modifications made to the model themselves, we can take a look together at how we can integrate those in the library.

Hi, thanks for your interest! From what I understand, I think your model falls in the category of dynamic acceleration. For these types of paper, I recommend you to integrate it to `examples/`, just like [PABEE](https://github.com/huggingface/transformers/tree/master/examples/bert-loses-patience) and [DeeBERT](https://github.com/huggingface/transformers/tree/master/examples/deebert). I've emailed you an invitation to our Slack channel if it works for you. cc @LysandreJik 

 @LysandreJik yes, that is correct. Thanks @JetRunner, let's discuss more on the slack.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I am sorry, but I have been fully loaded with some other stuffs. I won't be able to make a progress. I'd like to close this to avoid any confusion.
Add FastFormers to the example directory
Author: ykim362
Pull request: yes
Labels: name: WIP

# What does this PR do?

Add FastFormers into the example directory.
https://github.com/huggingface/transformers/issues/8083

https://arxiv.org/abs/2010.13382
https://github.com/microsoft/fastformers

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@JetRunner @LysandreJik 



Comments:
Thanks, @JetRunner ! I will clean up the code and fix the CI checks.

Thanks for the opinions, @LysandreJik!

1. The onnx related files are necessary before it's merges into master branch of onnxruntime
2. I agree with you regarding the binary files. I can put the binary files in a different place and put a script to download them, here.
3. That sounds good. I will look at the dataset library to see how to utilize it.

That's great, thanks a lot @ykim362! I think we can do with the ONNX files while your PR is in waiting over at onnxruntime.

Thanks for the review @patrickvonplaten ! I can make changes most of them as you recommended.

Regarding `attention_head_size`, this is necessary for head pruned transformers. I can think of two ways to keep backward compatibility.
1. Create a new model class by subclassing `RoBERTa`. FastFormers supports BERT and RoBERTa, so that would work for both.
2. In the current BERT model, add a default behavior (same as current logic) when `attention_head_size` doesn't exist. Then, it could be used only when `attention_head_size` parameter exists in the config file.

Or, I am open to any suggestions. :)

What is the status with this project? Anything I can help with?

> What is the status with this project? Anything I can help with?

I think it's mostly about this: https://github.com/huggingface/transformers/pull/8637/files?file-filters%5B%5D=.png&file-filters%5B%5D=.py&file-filters%5B%5D=.whl#r545051113 -> in a first PR we should not touch this logic IMO

I am sorry, but I have been fully loaded with some other stuffs. I won't be able to make a progress. I'd like to close this to avoid any confusion.

Thank you for the clarification @ykim362, I hope we may still collaborate in the future!

Thanks, @LysandreJik ! Likewise for the future collaboration!  :)
Add BlenderBot small tokenizer to the init
Author: LysandreJik
Pull request: yes

This class was forgotten, adding it to the init and the documentation.

Comments:
Actually this tokenizer seems a bit broken:
```py
tokenizer = AutoTokenizer.from_pretrained("facebook/blenderbot_small-90M")
```
```
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/home/lysandre/Workspaces/Python/transformers/src/transformers/models/auto/tokenization_auto.py", line 469, in from_pretrained
    return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/home/lysandre/Workspaces/Python/transformers/src/transformers/tokenization_utils_base.py", line 1741, in from_pretrained
    return cls._from_pretrained(
  File "/home/lysandre/Workspaces/Python/transformers/src/transformers/tokenization_utils_base.py", line 1858, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/lysandre/Workspaces/Python/transformers/src/transformers/models/blenderbot_small/tokenization_blenderbot_small_fast.py", line 76, in __init__
    ByteLevelBPETokenizer(
  File "/home/lysandre/transformers/.env/lib/python3.8/site-packages/tokenizers/implementations/byte_level_bpe.py", line 36, in __init__
    BPE(
Exception: Error while initializing BPE: Token `_</w>` out of vocabular
```

Which only appears after fixing with 6d90d5a3213b812f3d54d6142751a275b59343b1

cc @patil-suraj if I recall correctly you implemented this tokenizer, do you remember what might have gone wrong?
Finetuning BART on a  multi-input sequence to sequence task
Author: nbravulapalli
Pull request: no

I finetuned bart-base on a sequence to sequence task and I have the following questions:

a) Currently I structured the input and output for the bart model in "t5-style" by adding prefixes in front of each piece of input. For bart how should I give multiple inputs (or train it to return multiple outputs) to the model (is there special token to separate inputs, should I continue the t5-style prefixes, etc.)? Also, how would I do this for gpt-2/gpt-neo?

b) When finetuned with prefixes, the target data is formatted with "output: ......", however, the finetuned-bart returns "outputoutput: ......". Why is this repetiton occurring? Also, does the Bart tokenizer automatically add the eos token?

c) Also does the trainer api automaticallly handle ```adjust_logits_during_generation``` and ```decoder_start_token_id``` as discussed in this [post](https://discuss.huggingface.co/t/bart-base-rouge-scores/683)?

Could @patil-suraj or @patrickvonplaten help with this? This is my first project training an nlp model, and I would really appreciate any information you can offer regarding my questions.

Comments:
Hi there! It would be better if you post this on the [forum](https://discuss.huggingface.co/) instead since this is much general question and not an issue. You can tag me on the forum using `@valhalla` :) 

Use issues to report bugs or for feature requests. Thanks!


Thanks for your reply! I will close this issue and repost it on the forum.

EDIT:

@patil-suraj I have posted this on the huggingface forum [here](https://discuss.huggingface.co/t/finetuning-bart-on-a-multi-input-sequence-to-sequence-task/10201?u=nr1). Can you please take a look at it? Thank you!
Unable to execute RobertaModel from pretrained
Author: akku1506
Pull request: no
Labels: name: Migration

### Defined a class that inherits from RobertaPretrainedModel and concats image embedding

class XLMRobertaWithImageConcatenationMultiOutputClassifier(RobertaPreTrainedModel):

   def __init__(self,  config_bert , num_labels):

        super(XLMRobertaWithImageConcatenationMultiOutputClassifier, self).__init__(config_bert)
        self.num_labels = num_labels
        self.num_another_labels = num_another_labels
        self.bert = RobertaModel(config_bert)
        self.backbone = models.resnet50(pretrained = False)        
        num_features = self.backbone.fc.in_features
        self.img_hidden_size = 512
        self.pre_classify_hidden = 512        
        self.pre_classify_fc = torch.nn.Linear(config_bert.hidden_size + self.img_hidden_size, self.pre_classify_hidden)
        self.classifier = torch.nn.Linear(self.pre_classify_hidden, num_labels)
        self.apply(self._init_weights)



### Trying to call XLMRobertaWithImageConcatenationMultiOutputClassifier from pretrained
model = XLMRobertaWithImageConcatenationMultiOutputClassifier.from_pretrained( 'roberta-base', num_labels = NUM_CLASSES)

### Error
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-22-d0dbf641e2da> in <module>
---> 17 model = XLMRobertaWithImageConcatenationMultiOutputClassifier.from_pretrained('roberta-base',, num_labels=NUM_CLASSES)

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   1323         else:
   1324             with no_init_weights(_enable=_fast_init):
-> 1325                 model = cls(config, *model_args, **model_kwargs)
   1326 
   1327         if from_pt:

TypeError: __init__() missing 1 required positional argument: 'num_labels'

Is there something I am missing? Despite giving num_labels it still asks for that parameter? Any idea or help would be highly appreciated.


Comments: