noamaanMulla-03/transformers-issues huggingface dataset:
Bort (Amazon's reduced BERT)
Author: raulcarlomagno
Pull request: no
Labels: name: New model

# 🌟 New model addition

## Model description

Amazon Alexa researchers extract an optimal subset of architectural parameters for the BERT architecture by applying recent breakthroughs in algorithms for neural architecture search. The proposed optimal subset, “Bort,” is just 5.5 percent the effective size of the original BERT-large architecture (not counting the embedding layer), and 16 percent of its net size.

## Open source status
using mxnet and gluonnlp

paper https://arxiv.org/pdf/2010.10499.pdf
repo https://github.com/alexa/bort

* [X] the model implementation is available: (give details)
* [X] the model weights are available: (give details)
* [@adewynter] who are the authors: (mention them, if possible by @gh-username)


Comments:
Any update on this one?

This was added in #9112
Add POINTER model
Author: dreasysnail
Pull request: no
Labels: name: New model

# 🌟 New model addition

## Model description

[POINTER](https://github.com/dreasysnail/POINTER) is a progressive and non-autoregressive text generation pre-training approach, published on EMNLP 2020 by Microsoft Research. POINTER generates fluent text in a progressive and parallel manner. With empirical logarithmic time, POINTER outperforms existing non-autoregressive text generation approaches in hard-constrained text generation.

The model uses basically BERT-large architecture. However, an additional token is added to the vocab. The inference is performed by passing the input iteratively to the model. Since there is no existing model architecture in Huggingface that is compatible, I am not sure how to incorporate this into the model card. 

## Open source status

* [x] the model implementation is available: (https://github.com/dreasysnail/POINTER)
* [x] the model weights are available: [here](https://yizzhang.blob.core.windows.net/insertiont/ckpt.tar.gz?st=2020-08-18T20%3A49%3A02Z&se=2024-01-16T20%3A49%3A00Z&sp=rl&sv=2018-03-28&sr=b&sig=PKrSJt38cmY0P%2FBcZuyK%2Btm3bXyYzzfazaqTu1%2F%2FDtc%3D)
* [x] who are the authors: @dreasysnail 


Comments:
Thanks @patrickvonplaten for taking this. It's nice to work with you again :)

Really interesting approach :hugs: 

@dreasysnail Do you think it is possible to pre-train a model from scratch on **one** GPU in a reasonable time? Could you say something about your used hardware setup and training time for the pre-training phase :thinking: 

Thanks @stefan-it ! Regarding your question:

> @dreasysnail Do you think it is possible to pre-train a model from scratch on **one** GPU in a reasonable time? Could you say something about your used hardware setup and training time for the pre-training phase 🤔

The speed advantage of this algorithm is more on the decoding side. For the training time, you can expect this takes roughly similar amount of time comparing to, say, fine-tuning a BERT. One GPU is possible but if your dataset is large the training could be slow. So I would recommend you fine-tune from what we have already pretrained for fast convergence and better quality. 

For your reference, we were using 8/16*V100 GPUs to pretrain and fine-tune the models. The pretraining takes roughly one week and the fine-tuning takes 1-2 days.  



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Vision Transformer
Author: czabo
Pull request: no
Labels: name: New model

# 🌟 New model addition

## Model description

This issue is adding the Vision Transformer model described in the [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://arxiv.org/abs/2010.11929) paper. If you have any feedback and/or further ideas for the implementation please don't hesitate to mention them.

## Open source status

* [ x] the model implementation is available: 

[The official github repo](https://github.com/google-research/vision_transformer) provides the implementation in Jax/Flax.

* [x ] the model weights are available: 

See the github repo above.

* [ x] who are the authors: (mention them, if possible by @gh-username)

Google Research, Brain Team



Comments:
This was implemented in https://github.com/huggingface/transformers/pull/10950
Add CodeTrans, a model for source code generation, documentation generation and similar subtasks.
Author: tanmaylaud
Pull request: no
Labels: name: New model

# 🌟 New model addition

 Paper: https://arxiv.org/abs/2104.02443

<!-- Important information -->

## Open source status

* [ ] the model implementation is available: https://github.com/agemagician/CodeTrans
* [ ] the model weights are available:
https://github.com/agemagician/CodeTrans
Currently tensorflow.
* [ ] who are the authors: @agemagician 


Comments:
Hi @tanmaylaud ,

Thanks for creating this issue and your interest.

We already integrated our models into hugging face :)
You can find all of them here:
[https://huggingface.co/models?search=code_trans_](https://huggingface.co/models?search=code_trans_)

@patrickvonplaten : FYI, we have released the paper and all the models (146 models). All the models are fully integrated into hugging face library.

@agemagician Sorry, I didn't notice it may be because of the search terms I used. 
Thanks ! 
I think it deserves a separate category instead of 'Summarization'. 

Closed by @agemagician  :)
Routing Transformers / Add Google PG-19 Models
Author: GenTxt
Pull request: no
Labels: name: New model

# 🌟 New model addition - Google PG-19 Models

## Model description

Model checkpoints finally released as discussed in "Efficient Content-Based Sparse Attention with Routing Transformers'
Aurko Roy, Mohammad Saffar, Ashish Vaswani, David Grangier (https://arxiv.org/abs/2003.05997)

## Open source status

* [X ] the model implementation is available: (same link as below)
* [ X] the model weights are available: ( https://github.com/google-research/google-research/tree/master/routing_transformer)
* [X ] who are the authors: (see above)

Note: These tf2 models require proper conversion to pytorch versions and modifications to scripts to enable training and inference.


Comments:
There is an open-source pytorch implementation already - https://github.com/lucidrains/routing-transformer
Can't we adapt RT @lucidrains wrote to HF? 

I've checked the repo before and was hoping with the release of the models
this would be possible.

The original models may be tf1 and not tf2 format. This requires a custom
conversion script to pytorch.

Perhaps coders with advanced python skills will show interest in solving
the above issues.

On Wed, Jul 14, 2021 at 7:53 AM vblagoje ***@***.***> wrote:

> There is an open-source pytorch implementation already -
> https://github.com/lucidrains/routing-transformer
> Can't we adapt RT @lucidrains <https://github.com/lucidrains> wrote to HF?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/11686#issuecomment-879827190>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFMAWPKXKV5UXM2HFEQ57U3TXV3ERANCNFSM44WG7YGA>
> .
>
Fix special tokens not correctly tokenized
Author: qqaatw
Pull request: yes

# What does this PR do?

Fixes #13483.

I think this is a quite complex issue. The purpose of `unique_no_split_tokens` in `tokenization_utils.py` and `all_special_tokens` in `tokenization_utils_base.py` seems ambiguous as they all store special tokens that don't want to be splitted or lower_cased. This two separated variables may result in a hard maintainable code. I'm guessing this is probably due to the workaround for FastTokenizers' functionality.

Additionally, should we resolve this TODO?

https://github.com/huggingface/transformers/blob/c37573806ab3526dd805c49cbe2489ad4d68a9d7/src/transformers/tokenization_utils.py#L275-L282


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger @LysandreJik 

Comments:
A test has been added. Thanks for the review!
Removed console spam from misfiring warnings
Author: Rocketknight1
Pull request: yes

A major source of UX annoyance is that when our code is used with Keras, warnings about inappropriate arguments fire every time the model is traced/compiled, which is at least once, and sometimes several times, per training run. I don't know why the warnings were written like this in the first place, but they're definitely not working now, so I'm going to remove them for now and do a deeper dive into the reason for the misbehaviour when I have more time.

Comments:
The warning fires regardless of whether the options are set or not. I think it's only supposed to show when the user initializes a model, and then overrides those arguments when calling the model in graph mode. 

However, because of the way `inputs_processing` handles things the non-overridden inputs are set to the config values and not to `None` when the arguments are passed to `boolean_processing`, so it just fires (often several times) every time we `fit()` any Transformers model. This means that users just get used to ignoring the warning the whole time anyway.

I'm 99% sure we can just choose solution 1) and totally remove the warning with no problems - I'm not really sure why we have this restriction on changing parameters in graph mode anyway in modern Tensorflow!

> The warning fires regardless of whether the options are set or not.

Yes, like I said, I think it's because the top model resolves the kwargs passed, then passes them to the base model which also sends the thing to `input_processing`.

But if you are sure they are unnecessary, let's remove them **and** the change in the kwargs.

I'm still a little bit afraid of changing the actual input processing until I do a deep dive into everything! I made a change to the PR: The warning still exists, but it confirms that the overridden value is actually different from the config value before it logs the warning. This eliminates the spam in all cases except the ones when it was actually supposed to display in the first place.
top-k sampling for Flax models
Author: khalidsaifullaah
Pull request: no

# 🚀 Feature request

We're using a custom (`Flax`) Seq2Seq model (`Bart`) in our `DALL·E mini` project to generate the image tokens, currently, we're doing the following to generate samples (`encoded image`) given a tokenized prompt to the Seq2Seq model:
```
model.generate(
  **tokenized_prompt, 
  do_sample=True, 
  num_beams=1, 
  prng_key=subkey
)
```

But, now we're trying to experiment with the generation method (for example `top_k` sampling) to see if it improves our generated samples. 

We've tried doing the following:
```
model.generate(
        **tokenized_prompt,
        do_sample=True,
        top_k=50,
        prng_key=key,
        params=params
    )
```
Which throws- `NotImplementedError: Beam sampling is currently not implemented.`
After looking at the source-code for `generation_flax_utils`, it's indeed the case that it hasn't been implemented yet ([here](https://huggingface.co/transformers/_modules/transformers/generation_flax_utils.html)).

I hope the sampling feature will be integrated into the flax models, our project and other (Flax ones) will benefit greatly from the feature (and I'm sure it's already in the todo-list of the amazing HF dev team).
[_project repo_](https://github.com/borisdayma/dalle-mini)

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

We believe that the `top_k` sampling will improve our generations (quite a bit if not a lot), and even in the taming transformers paper, they've used `top_k` sampling for generating tokens with their autoregressive models and have shown higher `top_k` (subjective to the dataset) improves image generations by increasing variance (in a sense). So we just want to experiment with this sampling method and see if we can find something similar for our model.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Use `config_dict_or_path` for deepspeed.zero.Init
Author: aphedges
Pull request: yes
Labels: name: DeepSpeed

# What does this PR do?

While #13587 fixes one DeepSpeed warning, it missed a similar one. This applies the same change of `config` -> `config_dict_or_path` in another location.

## Who can review?

@sgugger (same reviewer as #13587)

Comments:
I'll let @stas00 merge this after he double-checks it's good :-)

@stas00, you're welcome! Thanks for the review!
fix some docstring in encoder-decoder models
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix a few places in some encoder-decoder docstrings. For example,

`:class:~transformers.EncoderDecoder is a generic ...` -> `:class:~transformers.SpeechEncoderDecoderModel is a generic ... `
in `modeling_speech_encoder_decoder.py`.

## Who can review?

@patrickvonplaten 


Comments:
Fix tests without any real effect in EncoderDecoderMixin
Author: ydshieh
Pull request: yes

# What does this PR do?

In `test_modeling_encoder_decoder.py`, there are 2 places like

```
enc_dec_model.save_pretrained(tmpdirname)
EncoderDecoderModel.from_pretrained(tmpdirname)

after_outputs = enc_dec_model(...)
```
Therefore `after_outputs` will be exactly the same `outputs`, since `enc_dec_model` doesn't get the new value.
For the testing purpose, we need to do
```
enc_dec_model = EncoderDecoderModel.from_pretrained(tmpdirname)
```
(Hope I get it right)

## Who can review?

@patrickvonplaten @sgugger 


Comments:
Thanks a lot @ydshieh !

The model should have been cast to the `torch_device` as otherwise the inputs/model are on different devices :)

https://github.com/huggingface/transformers/runs/3524847938?check_suite_focus=true

I'll take care of it :-) 

Hey, thank you guys! Sorry about the device issue, I am less familiar with PyTorch, and it seems the checks I had when I opened this PR don't have that test above (probably it only runs when we merge to master?)

Don't worry about it @ydshieh :-)
Fixes issues with backward pass in LED/Longformer Self-attention
Author: aleSuglia
Pull request: yes

## Description 

This PR fixes the computational graph created when computing the global attention scores in LED/Longformer Self-attention. The current implementation breaks the computational graph preventing the model from running the backward pass correctly in some cases. As explained in the dedicated issue, this problem arises in the current version of PyTorch (PyTorch 1.9) but not in one of the previous ones (PyTorch 1.7.1). 
This PR simply clones the appropriate tensors in order to avoid the issue. `clone()` is a differentiable operation so there are no changes to the actual model behaviour.

The issue is carefully reproduced in the following Google Colab: https://colab.research.google.com/drive/13rKxs6Ype0kDEBlnywsGynE2zpzv2CR-?usp=sharing

Fixes #12613

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ibeltagy @patrickvonplaten 


Comments:
RAG: building my own dataset
Author: Arij-Aladel
Pull request: no

@shamanez Hello! My question is about building our own dataset to train RAG from scratch. It was mentioned [here](https://github.com/huggingface/transformers/tree/master/examples/research_projects/rag#finetuning) that we can build our dataset to finetune the model, but the links do not open. Could you please provide us with the steps and examples?

Comments:
Check this script.

https://www.github.com/huggingface/transformers/tree/master/examples%2Fresearch_projects%2Frag%2Fuse_own_knowledge_dataset.py

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
やからん
Author: bdr2486
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
I modified https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py script few days ago for training electra from scratch. But there were some problems(maybe bugs) i had to solve for this task.
Author: apkbala107
Pull request: no

I modified https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py script few days ago for training electra from scratch. But there were some problems(maybe bugs) i had to solve for this task. 

Currently I´m setting up a clean running version for training a electra language model from scratch with an additional document classification head based on the script.

_Originally posted by @miketrimmel in https://github.com/huggingface/transformers/issues/4425#issuecomment-630715171_

Comments:
location is currently not available...please share the exact location

Here's the folder containing the language modeling scripts: https://github.com/huggingface/transformers/tree/master/examples/pytorch/language-modeling

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
location is currently not available...please share the exact locationDetailed Explanation
Author: apkbala107
Pull request: no

Detailed Explanation
https://mlcom.github.io/Create-Language-Model/

_Originally posted by @mlcom in https://github.com/huggingface/transformers/issues/4425#issuecomment-774689668_

Comments:
location is currently not available...please share the exact location

@apkbala107 mlcom.github.io

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[Vision Transformers] examples and pretraining
Author: flozi00
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

Example scripts for fine-tuning and pretraining for clip and Beit models

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

At the moment I am training ViT and DeiT for several tasks and thought it would be interesting to compare with clip or from scratch pretrained beit model in self supervised way.
I saw there are already examples for training clip using transformers and flax, not sure if there is an specific reason why it's not already in this repo.

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->

If there is someone out who could give me advice, I would be interested adding the Beit pretraining script for pytorch. Longtime I definitely want to add comparisons for several tasks between ViT and from scratch trained BeiT especially for low resource tasks to the hub

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[WIP] Add classification head for T5 and MT5
Author: PhilipMay
Pull request: yes

This is an experiment to add a classification head for T5 and MT5 models.
It uses the `T5EncoderModel` and `MT5EncoderModel` without the decoder part.

Comments:
@LysandreJik or @sgugger 
I am trying to add a classification head on top of the T5 and MT5 encoder models.
It works but the performance (metric) of my downstream tasks is very low compared to other models.
Could you maybe do a very quick code review on this PR please?

Maybe you find obvious bugs. 

Is the initialization maybe wrong for example?

I am adding a `SequenceSummary` head like XLNet is doing.
This has different values for `summary_type`.
I tried everything: `first`, `last`, `cls_index`. Noting realy works good.


PS: T5 and MT5 do not have these `
` or `[CLS]` token...


Hi @LysandreJik or @sgugger ,
I would be super happy if you could maybe give me some pointers here.

Maybe @patrickvonplaten will have some ideas, but maybe it's just that those types of models are not really good for sequence classification.

Hmm, T5 was really made so that every task (including classification tasks) are framed as a text2text task. I don't really see why one would use T5 over BERT for a "Encoder-only" classification task...Could you give a bit more detail on the motivation of this PR? :-) @PhilipMay 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.