noamaanMulla-03/transformers-issues huggingface dataset:
Wiki content part : XLM-RoBERTa,   "xlm-roberta-base"
Author: pangbochen
Pull request: no

In the wiki page for XLM-R
https://huggingface.co/transformers/model_doc/xlmroberta.html

the model name should be 'xlm-roberta-base'  instead of 'roberta-base'

@sgugger 

Comments:
Yes, that's because those models are direct subclasses of Roberta, so they have the same documentation.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
how to continue pre-train custom data
Author: moseshu
Pull request: no

1.problem
   I want to pretrain my specific data, reset the numbers of the encoders or layers.
can you offer an interface to load our custom data which is like json or other type to 
pretrain. the LMASK, NSP,PE is packaged in the interface.



Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Fix examples in VisualBERT docs
Author: gchhablani
Pull request: yes

This PR fixes the examples in [VisualBERT docs](https://huggingface.co/transformers/master/model_doc/visual_bert.html).
The issue is described in this [comment](https://github.com/huggingface/transformers/pull/10534#issuecomment-853015418) by @NielsRogge.

Requesting review from @sgugger

Comments:
[WIP] FNet
Author: gchhablani
Pull request: yes

This PR adds FNet in PyTorch.

- Paper: https://arxiv.org/pdf/2105.03824v2.pdf
- Code and Checkpoints: https://github.com/google-research/google-research/tree/master/f_net
- Authors: @jtainslie @ilyaeck @santiaontanon-google


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I am still working on this PR in a branch and will create another PR when it's somewhat ready.
Error in the model card page
Author: yanan1116
Pull request: no

https://huggingface.co/facebook/bart-large-mnli
this page is about the 

> facebook/bart-large-mnli model. 

However, in the section of 

> With manual PyTorch


the example show the loaded model is joeddav/xlm-roberta-large-xnli, which is not 

> facebook/bart-large-mnli

I suppose that this is an error .

Comments:
Thank you for the report, I have updated the modelcard in [`hf.co#c626438`](https://huggingface.co/facebook/bart-large-mnli/commit/c626438eeca63a93bd6024b0a0fbf8b3c0c30d7b)
Fix TYPE_CHECKING not imported
Author: minwhoo
Pull request: yes

# What does this PR do?

Fixes omitted import of TYPE_CHECKING in xlm_prophetnet model code's __init__.py.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
Actually the whole init needs to be rewritten to be the same as other models, it was somehow missed when we converted all models. Would like to amend your PR in that direction?

Your rebase has introduced many file changes in the diff that make the PR unreadable. Once you're satisfied with your branch, could you close this PR and open a new one? This should make the diff better.

Sure thing.
Any example to accelerate BART/MBART model with onnx runtime？
Author: junleiz
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Transformers.onnx provided a simple way to convert the bart/mbart model to onnx model. But there is no example about how these model could be actually exc by onnx runtime. Bucause some onnx models may occur various of errors. Could you please provide some examples about using onnx runtime to exc these models ?


Comments:
The transformer.onnx provides a simple to convert hf models to onnx models. But there is no examples about how to exc them by onnx runtime. Some onnx models actually would meet some errors exc with onnx runtime. Could you please provide some examples and experiments about running them with onnx runtime?
Is there any way to train a transformer model from scrat
Author: Hannibal046
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Comments:
How to construct a pretrain by myself using Tensorflow2 +Keras?
Author: Chin-Yi-Ji
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Comments:
In case you want to pre-train a model on your own data in Tensorflow, please take a look at the [Tensorflow examples](https://github.com/huggingface/transformers/tree/master/examples/tensorflow/language-modeling).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
How to fine-tune a model with my custom tokenizer?
Author: Lucas-Luoling
Pull request: no

1. I want to train a tokenizer with my own datasets.
2. I have no enough data and resources to train a pretrained model from scratch.

pipeline:
1. train a tokenizer like [tokenizer_training](https://github.com/huggingface/notebooks/blob/master/examples/tokenizer_training.ipynb)
2. do something with current pretrained model, get a new pretrained model
3. fine-tune the pretrained model with my domain datasets, get my domain pretrained model
4. fine-tune the domain pretrained model for my downstream task.

and what should I do in the 2rd step?

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[Pegasus][tokenizer] pegasus tokenizer doesn't have any BOS token?
Author: sajastu
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.9.0.dev0
- Platform: Ubuntu
- Python version: 3.8
- PyTorch version (GPU?): 1.7 (yes)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

@patrickvonplaten, @patil-suraj, @LysandreJik


## Information

Model I am using (PEGASUS ...):

This is not specifically a bug in transformers, but I'm wondering if I can comment and ask my question in this section as there was no other section I could relate this to.

I'm trying to use pegasus's encoder for extractive summarization proposes so that system can identify top sentences. To do so, I want to implement an approach like BERTSUM where [CLS] tokens are being added to the beginning of each sentence. 

Talking of pegasus, I noticed there's no so-called BOS (similar to CLS head in BERT) token which denotes the start of each sentence (below) –although it's available in BART's tokenizer.

https://github.com/huggingface/transformers/blob/master/src/transformers/models/pegasus/tokenization_pegasus.py#L102

```
...
def __init__(
        self,
        vocab_file,
        pad_token="<pad>",
        eos_token="
",
        unk_token="<unk>",
        mask_token="<mask_2>",
        mask_token_sent="<mask_1>",
        additional_special_tokens=None,
        offset=103,  # entries 2 - 104 are only used for pretraining
        sp_model_kwargs: Optional[Dict[str, Any]] = None,
        **kwargs
    ) -> None:
...
```

However, there's `additional_special_tokens` by which I can define new special tokens (like the one I intend to use for sentences). Is that possible? I mean, is the new token expected to give representations of what tokens (i.e.,. sentences) following it? Thanks!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Deit
Author: HamidShojanazeri
Pull request: no

# 🚀 Feature request

The Deit model from torchhub is scriptable, however the pertained version in HF repo, "facebook/deit-base-distilled-patch16-224" is the distilled one which is not scriptable. 

This can be tried out using the code below 
```
import torch
model = torch.hub.load('facebookresearch/deit:main', 'deit_base_patch16_224', pretrained=True)
ts_model = torch.jit.script(model)
```
vs 

```
feature_extractor = DeiTFeatureExtractor.from_pretrained('facebook/deit-base-distilled-patch16-224')
model = DeiTForImageClassificationWithTeacher.from_pretrained('facebook/deit-base-distilled-patch16-224')
ts = torch.jit.script(model)
```
## Motivation
Torchscripted models can boost the inference speed and it would benefit the community having vision models that are scriptable.




Comments:
Are you interested in working on this?

Basically, to make DeiT scriptable, you need to set `test_torchscript` of the model test as seen [here](https://github.com/huggingface/transformers/blob/0d1f67e651220bffef1441fa7589620e426ba958/tests/test_modeling_deit.py#L160) to `True`, then run the tests (running `pytest tests/test_modeling_deit.py` from the root of the repo), and then fix the modeling file (`modeling_deit.py`) to make the failing tests pass.

We could then do the same for `modeling_vit.py`. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
XLM tokenizer lang2id attribute is None
Author: Quang-Vinh
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.5.1
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.8
- PyTorch version (GPU?): 1.8.1+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: No

### Who can help

Models:

- albert, bert, xlm: @LysandreJik

Library:

- tokenizers: @LysandreJik

## Information

Model I am using XLM with Causal language modelling:

The problem arises when using:
* [x] the official example scripts: (give details below)

## To reproduce

Steps to reproduce the behaviour:

1. Run example code from https://huggingface.co/transformers/multilingual.html 

``` python
import torch
from transformers import XLMTokenizer, XLMWithLMHeadModel

tokenizer = XLMTokenizer.from_pretrained("xlm-clm-enfr-1024")
model = XLMWithLMHeadModel.from_pretrained("xlm-clm-enfr-1024")

language_id = tokenizer.lang2id['en']
```

The attribute lang2id is None and so I get a Nonetype is a non-suscriptable error. Following the example I am expecting to get 0 for language_id.

As a side note, it says that these checkpoints require language embeddings which I'm assuming is from the argument langs. What is the default behavior when this is not provided? I tried looking at https://huggingface.co/transformers/glossary.html but could not find any reference to it.


Comments:
FYI, I tried downgrading and I found that the most recent version that doesn't have this bug is `transformers==4.3.3`. So you could try downgrading to that version for now, until someone fixes it.

```
pip install transformers==4.3.3
```

Thanks for the advice! So far I have just been manually specifying the language id for the two languages, hopefully, that is sufficient as well.

Hello! Sorry for taking so long to get back to this issue - the issue should normally be fixed now, for all versions. We updated the configurations of the XLM models on the hub.

Thanks for flagging!

Hi @LysandreJik is the update going to solve this XLM issue as well? https://github.com/huggingface/transformers/issues/12174

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Weird outputs by `opus-mt-en-es`
Author: bhavitvyamalik
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: TPU VM
- Python version: 3.8.10

### Who can help
@patrickvonplaten @patil-suraj 

## Information
Model I am using: FlaxMarianMTModel


## To reproduce
I used this code for beam size 2 and 4. Funnily the outputs looked almost same just that Oh changed to no in beam_size=2

```
from transformers import MarianTokenizer, FlaxMarianMTModel
model = FlaxMarianMTModel.from_pretrained('Helsinki-NLP/opus-mt-en-es', from_pt=True)
tokenizer = MarianTokenizer.from_pretrained('Helsinki-NLP/opus-mt-en-es')
text = "Living Room, The Sheridan House! Your Minneapolis Home!"
input_ids = tokenizer(text, max_length=64, return_tensors='jax', truncation=True)
sequences = model.generate(**input_ids, early_stopping=True, max_length=64, num_beams=2).sequences
tokenizer.batch_decode(sequences, skip_special_tokens=True, max_length=64)
```
For num_beams = 2 output:
'Sala, ¡La Casa Sheridan, tu hogar de Minneapolis, ¡No, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no'

For num_beams = 4 output:
'¡Sala de estar, la Casa Sheridan, tu hogar de Minneapolis, ¡Oh, ¡Oh, oh, oh, oh, oh, oh, oh, oh, oh, oh, oh, oh, oh, oh, oh, oh, oh, oh, oh, oh,'

## Expected behavior
Shouldn't give 'oh' or 'no' in outputs.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Fixed via #12662
MT5-base tokenizer can't decode to target language after decoding
Author: pranonrahman
Pull request: no

## Environment info

- `transformers` version: 4.9.0
- Platform: Google Colab
- Python version: 3.8+
- PyTorch version (GPU?):  1.9.0+cu102
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help


Models:

- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj


Library:

- tokenizers: @LysandreJik


## Information

Model I am using (MT5):

The problem arises when using:
* [ ] my own modified scripts: When I am finetuning mt5-small model for question answering using mt5ForConditionalGeneration, after running inference, the output is not in the specified language. 

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (Question Answering)

## To reproduce
link to my notebook: [link](https://colab.research.google.com/drive/12nMMdHul4Avxn38o3LZhsVgVE02I6g2E?usp=sharing)
Steps to reproduce the behavior:

1. Run the inference section
2. Run on any language
3. The model outputs in a mixed language

## Expected behavior
The expected behavior should be to produce output on a single language.


Comments:
Add try-except for torch_scatter
Author: JetRunner
Pull request: yes

Add an error message for the CUDA version mismatch of `torch_scatter`.



Comments:
I don't think this will work. It's a `RuntimeError` being raised by `torch_scatter`, not an `OSError`. See the specific code at [line 59 of `__init__.py`](https://github.com/rusty1s/pytorch_scatter/blob/2.0.8/torch_scatter/__init__.py#L59). Also, this replaces the existing informative error message from `torch_scatter` with a less informative one.

@aphedges Thanks for the note - I have edited the description that does not indicate the association with your issue anymore. Also, the intention of this PR is to simply circumvent the error since in most cases, people just don't use TAPAS but still get blocked by this error.

Also, the original `torch_scatter` error message is not informative at all. It just says some file cannot be located and after some googling, I realize it's due to the CUDA version. So I'm basically replacing that with my googled solution.

@JetRunner, thanks for editing the description!

Sorry about my note about `RuntimeError` vs. `OSError` earlier. I think I got confused by the fact that `torch-scatter` explicitly throws a runtime error for some CUDA version mismatches, but the error you're logging here is for a different CUDA version mismatch that doesn't have a good error message. I think I had to google this one, too, so your error message is definitely an improvement.
Remove usage of local variables related with model parallel and move …
Author: hyunwoongko
Pull request: yes

# What does this PR do?
This PR is related with [model parallel integration from Parallelformers](https://github.com/huggingface/transformers/issues/12772).
You can check detail of PR here: https://github.com/tunib-ai/parallelformers/issues/11#issuecomment-894719918


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@stas00 


Comments:
Most of these modifications are encoder-decoder models from Bart's code and encoder models that has token type id. As you said, it is difficult to work on all models at once, so I will exclude the case where the model needs to be modified. I also agree that modifying multiple models at the same time makes it difficult to test. First, let's start with one decoder model like GPT-Neo. I will close this PR and upload a new one soon.

One more note: besides the dozens of models we also have a template. In this case it's mostly: https://github.com/huggingface/transformers/blob/master/templates/adding_a_new_model/cookiecutter-template-%7B%7Bcookiecutter.modelname%7D%7D/modeling_%7B%7Bcookiecutter.lowercase_modelname%7D%7D.py so when all is happy here, please let's not forget to apply the changes there as well.


I would like an approach that that does one model first, so we can clearly comment on the design, then all models after (unless it's very different for each model in which case, similar models by similar models if that makes sense).

As for the changes in themselves, I would need a clear explanation as to why the `token_type_ids` device need to be changed from the position_ids device. That kind of code should not be present in the modeling files as is, as people adding or tweaking models won't need/understand it. We can abstract away things in `PreTrainedModel` as you suggest @stas00, that seems like a better approach. Or maybe a method that creates those `token_type_ids` properly, at the very least.