noamaanMulla-03/transformers-issues huggingface dataset:
Enable option for subword regularization in `XLMRobertaTokenizer`
Author: PhilipMay
Pull request: yes

# What does this PR do?
I would like to use [subword regularization](https://github.com/google/sentencepiece#subword-regularization-and-bpe-dropout) from [google/sentencepiece](https://github.com/google/sentencepiece). The reason is that it might be used to improve downstream task performance.

Since `XLMRobertaTokenizer` already uses `SentencePieceProcessor` from `google/sentencepiece` there are only some minor modifications needed.

3 additional parameters are added to the constructor of `XLMRobertaTokenizer`. This are:
```python
        enable_sampling=False,
        nbest_size=-1,
        alpha=0.1,
```
The default values are selected so that this is no breaking change.

In the `_tokenize(self, text)` function there was a call to `self.sp_model.EncodeAsPieces(text)`. This call does ignore the parameters for subword regularization. That is why it had to be replaced by a call to `self.sp_model.encode(text, out_type=str)`.

Since `XLMRobertaTokenizerFast` is an independent implentation which does not use `google/sentencepiece` it is not in the scope of the PR to add subword regularization to the fast tokenizer.

## To-do
- [x] check if tests pass
- [x] check if tests can / should be added
- [x] add a link to a page where we can see all kwargs

## Who can review?
@LysandreJik @stefan-it 

Comments:
It would be awesome if you ( @LysandreJik and @stefan-it ) could give some feedback on this - although some tests still fail.
Is it a good idea? Would you merge it if everything is cleaned up?

I added a test, everything is green and IMO ready for review. @LysandreJik @stefan-it

@LysandreJik and @n1t0 I think that would be a good idea but IMO it should not be done in the scope of this PR.

Because the slow tokenizer just delegates the work to [google/sentencepiece](https://github.com/google/sentencepiece) this PR is very easy but adding that to the Rust Tokenizer would be way more work afaik.

Hey @LysandreJik and @n1t0 
I think this PR is somehow stuck...
AFAIK my change is ok for you. 
What about merging it and moving the part for the fast tokenizer to a seperate issue?

@LysandreJik 
- Requested changes are made and marked as resolved. 
- Inline questions are answered and marked as resolved.
- CI is green.

IMO ready for merge.

@sgugger all green again :-)

> Perfect! If you feel up to the task, I think all (slow) sentencepiece-based tokenizers could benefit from this addition.

see #11417
[Example] Fix a NaN bug in the flax mlm example
Author: merrymercy
Pull request: yes

## What does this PR do?

Fix a NaN bug in the flax masked language model example. This is a bug introduced in #9133
The min should be max. Otherwise, we will get a NaN.

## Who can review?
@TevenLeScao  @mfuntowicz 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

cc @patrickvonplaten 

Hey @merrymercy - super sorry, I saw the PR too late and it was actually already fixed.

Thanks for your effort on Jax integration! @patrickvonplaten 
Could you also add some doc for these examples https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling?
(Load dataset failure) ConnectionError: Couldn’t reach https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/cnn_dailymail/cnn_dailymail.py
Author: AI678
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->
Hey, I want to load the cnn-dailymail dataset for fine-tune.
I write the code like this
from datasets import load_dataset

test_dataset = load_dataset(“cnn_dailymail”, “3.0.0”, split=“train”)

And I got the following errors.

Traceback (most recent call last):
File “test.py”, line 7, in
test_dataset = load_dataset(“cnn_dailymail”, “3.0.0”, split=“test”)
File “C:\Users\666666\AppData\Local\Programs\Python\Python38\lib\site-packages\datasets\load.py”, line 589, in load_dataset
module_path, hash = prepare_module(
File “C:\Users\666666\AppData\Local\Programs\Python\Python38\lib\site-packages\datasets\load.py”, line 268, in prepare_module
local_path = cached_path(file_path, download_config=download_config)
File “C:\Users\666666\AppData\Local\Programs\Python\Python38\lib\site-packages\datasets\utils\file_utils.py”, line 300, in cached_path
output_path = get_from_cache(
File “C:\Users\666666\AppData\Local\Programs\Python\Python38\lib\site-packages\datasets\utils\file_utils.py”, line 475, in get_from_cache
raise ConnectionError(“Couldn’t reach {}”.format(url))
ConnectionError: Couldn’t reach https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/cnn_dailymail/cnn_dailymail.py

How can I fix this ?
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
I got a same problem as you. I install the transformer from source but still has this problem. @sgugger 

I had the same problem, but this is how I solved it
1、Access the address and download the file：
https://raw.githubusercontent.com/huggingface/datasets/1.6.0/datasets/bookcorpus/bookcorpus.py
2、Put the file in this position：
![image](https://user-images.githubusercontent.com/58463989/115979713-09180000-a5ba-11eb-932a-440346f6e7ab.png)
3、Then you can load the corpus normally. I hope it can help you
IBert: What would be the possible reason `IntLayerNorm` does not decrease the loss?
Author: kyoungrok0517
Pull request: no

### Who can help
@kssteven418 

## Information
IBert

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## Problem
Hello. I'm trying to use `IntLayerNorm` in my model. The model without the layer trains properly, but if I add the layer as follow the model suddenly stops being learned (loss does not change). What would be the possible reason?

```python
def forward(self, x, k: int = None):
        x, scaling_factor = self.pre_linear_act(x)                                # QuantAct
        x, scaling_factor = self.linear(x, scaling_factor)                     # QuantLinear
        x, scaling_factor = self.post_linear_act(x, scaling_factor)      # QuantAct

        # normalize
        if self.normalize:
            x, scaling_factor = self.layer_norm(x, scaling_factor)               # IntLayerNorm
            x, scaling_factor = self.post_layernorm_act(x, scaling_factor) # QuantAct
```

## Update
I observed that the output tensor of `self.layer_norm` is all zero.

Comments:
It is likely that the IntLayerNorm layers are not warmed up. IntLayerNorm layer has to adjust its internal parameter (`self.shift`) during the quantization-aware training process. It is the one that prevents overflow (i.e. keeps the internal activation values to be less than 2**32) and is initialized with zero. Here is the relevant part in the code: https://github.com/huggingface/transformers/blob/master/src/transformers/models/ibert/quant_modules.py#L508

Therefore, if you skip the quantization-aware training process and immediately use the model for inference, those layers may produce some unexpected outcomes. Could this be your case?

Yeah I'm not using quantization-aware training so that'll be the reason. Thanks for the answer!
Add head_mask, decoder_head_mask, cross_head_mask to ProphetNet
Author: stancld
Pull request: yes
Labels: name: wontfix

This PR implements `head_mask`, `decoder_head_mask` and `cross_head_mask` for ProphetNet (and Longformer as there's a copy dependency) and it is the follow-up to the open issue #9814. 

**Motivation:** This PR is a part of an endeavour to enable the usage of `head_mask` and `decoder_head_mask` for all encoder-decoder transformers following the recent work on BART-like models (#9569).

<hr>

Fixes: https://github.com/huggingface/transformers/issues/9814

Reviewers: @patrickvonplaten

Comments:
This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.

Reopened as this might still be in the works.

This PR is required for #10605.

Also, it is necessary to rebase this branch to the current `master` [As a lot of changes have been done to the repo, there are some conflicts I'm gonna handle asap.].

Hi @LysandreJik - I fixed `cross_head_mask` for this `ProphetNetModel`. At this moment, there is an error regarding the `test_forward_signature` and there is likely to be a problem with a template. These issues should be then resolved in #10605 which takes care of `cross_head_mask` for all other encoder-decoder models which have already had `head_mask` and `decoder_head_mask` merged into the master.

Update: #10605 now passes all the tests. (@LysandreJik)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Super sorry for being incredibly slow here @stancld ! I think we can actually merge this if it passes all the tests :-)

@patrickvonplaten No worries, it's completely okay! :) I rebase this branch and now all the tests have passed.
`resize_token_embeddings` not taken into account in `save_pretrained` for `EncoderDecoderModel`
Author: rahular
Pull request: no

## Environment info
- `transformers` version: 4.5.0
- Platform: Darwin-17.7.0-x86_64-i386-64bit
- Python version: 3.7.2
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
@patrickvonplaten, @patil-suraj


## Information
I am extending the embeddings of the decoder of an `EncoderDecoderModel` model. When I save it, the config does not reflect the new size. However, it works fine when I try doing the same for non `EncoderDecoderModel` models.

## To reproduce
```
In [1]: model = t.EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-uncased', 'bert-base-uncased')

In [2]: model.decoder.bert.embeddings.word_embeddings
Out[2]: Embedding(30522, 768, padding_idx=0)

In [3]: model.decoder.resize_token_embeddings(30522+100)
Out[3]: Embedding(30622, 768)

In [4]: model.save_pretrained('test-bert')
```



## Expected behavior
The updated embedding size should be saved in `config.json`


Comments:
This is caused by the EncoderDecoderConfig which initializes independent objects ([link](https://github.com/huggingface/transformers/blob/d9c62047a8d75e18d2849d345ab3394875a712ef/src/transformers/models/encoder_decoder/configuration_encoder_decoder.py#L84)) instead of utilizing the already existing ones.

You can fix that for the moment by calling:
```
model.config.decoder = model.decoder.config
model.config.encoder = model.encoder.config
```
PR will follow.
Adding end-to-end retriever training to RAG with RAY implementation. 
Author: shamanez
Pull request: no

# 🚀 Feature request


Use of RAY to run separate processes for retrieve document indexes, training the system, and re-initialize the indexes with an updated context encoder.

## Motivation
Recent [papers](https://arxiv.org/abs/2101.00408) have shown that fine-tuning the entire retriever gives huge gains for QA tasks. Also having able to fine-tune end-end manner can give better results in different domains.


The idea is to train the RAG as is, but we keep updating the gradients of the context encoder with the supervised loss function (doc score mentioned in the RAG). Then in every n-steps, we re-initialize the embedding and indexes with updated context encoder weights. 


REALM [does this with a back-and-forth process ](https://github.com/google-research/language/tree/master/language/realm#running-the-code)which is only run on a single GPU.


As discussed in this [issue](https://github.com/huggingface/transformers/issues/9646#issuecomment-775309123), @lhoestq  suggested it would be easier to complete this with RAY since we can have separate actors. 


@richardliaw
@amogkam 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Support for DeBERTa V2 models
Author: saichandrapandraju
Pull request: no

Hi,

I downloaded [DeBERTa V2-XLarge](https://github.com/microsoft/DeBERTa) from [here](https://huggingface.co/microsoft/deberta-v2-xlarge) and trying to implement V2-XLarge model but I'm getting this error -

**RuntimeError: Error(s) in loading state_dict for DebertaForSequenceClassification:
	size mismatch for deberta.encoder.rel_embeddings.weight: copying a param with shape torch.Size([512, 1536]) from checkpoint, the shape in current model is torch.Size([1024, 1536]).**

I saw that vocab got changed for V2 models. If that is the reason for above issue, is there any workaround to implement V2 models with HF ?

Comments:
Hi, no workaround, we're working on the implementation now (https://github.com/huggingface/transformers/pull/10018). It should be available in a few days.

Thanks @LysandreJik 

@saichandrapandraju the PR was merged, so I think this issue can be closed now?

ok @yaysummeriscoming ,

May I know when will be the stable release( I think 4.4.0) for these merges?

Yes, this issue can be closed! v4.4.0 should be released in the next two weeks.

Has this been fixed? I'm downloading the model direct from huggingface [here](https://huggingface.co/microsoft/deberta-v2-xlarge) and i still get this error thrown

Are you using the `DebertaV2Model` or the `DebertaModel` ?

@LysandreJik I am using the simpletransformers library, i'm not sure if you're familiar with it but i believe by default it uses the DebertaModel, not sure how and if i can change it to DebertaV2Model
Differences between S2T and Wav2Vec2
Author: Narsil
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

There seem to be differences between S2T and Wav2Vec2 which are hard to understand reason about that may be fixable.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

Adding something like AutomaticSpeechRecognitionPipeline might be desirable and would be hard to do in current state. If/When new multimodal models are added it is going to add more and more complexity. Aiming for consistent API is desirable IMO.

## Description

- There are no `AutoProcessor.from_pretrained`.
- Wav2Vec2 cannot use `skip_special_tokens` for the decode variant (because it skips `<pad>` tokens early leading to removing all duplicate letters from the output. IMO it's a "bug" as `<pad>` in the context of CTC is not a special_tokens (at least until letters are resolved). 
- Wav2Vec2 uses 1 forward pass, where S2T uses  generate function. It would be nice, if there could be 1 interface only. (maybe just overload the `Wav2Vec2ForCTC.generate` ?
- S2T overloads `input_ids` with float tensors when `generating`, which works in practice but does seem like a piggy-back of the generate function and is definitely confusing to use. If `generate` is generic enough maybe then `input_ids` should be renamed to reflect that (`input_ids` are IDs everywhere in the rest of transformers. It could be a simple internal variable rename, I don't imply we should change any function signature anywhere, just that the variable is not necessarily IDs. Isn't it a bit like `inputs_embed` ?
- Wav2Vec2Processor returns 'input_values' where S2TProcessor returns 'input_features'. They seem (at least in appearance) to be the same. Would it be better to use only 1 name if they do ?

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->

Happy to contribute with PRs but I lack the more general view to be sure about what direction to take, and where are the "better" fixes.

Comments:
@patrickvonplaten 
Who should I tag for S2T ?

@patil-suraj for s2t

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
TokenClassificationPipeline: top-k predictions
Author: francescorubbo
Pull request: no

# 🚀 Feature request

Optional argument for TokenClassificationPipeline to output top-k predictions instead of limiting output to argmax.

## Motivation

Having access to the top-k prediction distribution is useful in a number of scenarios, such as confidence calibration (https://arxiv.org/abs/1706.04599) or generating pseudo-labels (https://arxiv.org/abs/1911.04252).

## Your contribution

I'm happy to submit a PR with the proposed changes, if this contribution is deemed useful.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Fix log message for training from checkpoint with global step
Author: ioana-blue
Pull request: no

# 🚀 Feature request

I think the log message here is wrong:

https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py#L978

I think it should read something along these lines:

```
logger.info(
                    f"  Will skip the first {epochs_trained} epochs then the first {steps_trained_in_current_epoch} "
                    "batches in the current epoch {epochs_trained + 1}."
                )
```

The batches that are skipped are not skipped in the first epoch since the training was already done for `epochs_trained`. As the variable name indicates, the training is going to skip the steps already trained in the current epoch. The current epoch is `epochs_trained + 1`. 

## Motivation

The log message was confusing. I went and traced the code to ensure that it does the right skipping (which I think it does). 

Comments:
Yes, that sounds clearer. Would you mind making a PR with this?

Will make time this week. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Train tokenizer for Deberta
Author: avacaondata
Pull request: no

Hi, I would like to know how can I train a DeBERTa tokenizer. From the paper I saw it uses BPETokenizer, but the BPETokenizer from huggingface/tokenizers doesn't work for this. Could you recommend me another implementation or library or a correct configuration for huggingface/tokenizers implementation to be able to train a DeBERTa model from scratch?


Comments:
HuggingFace has another library called [tokenizers](https://github.com/huggingface/tokenizers)  especially for this.

Currently, the training of Deberta Tokenizer is not supported directly by huggingface. Of course, you can create the required files by yourself from BPETokenizer training output, but you could also simply wait until #10703 is merged into the master branch and released. :-)

How would be the process of creating the required files from the BPETokenizer training output? @cronoik I'd really appreciate a little bit of explanation, as I tried to do so and I failed.

You can save me a lot of time by simply using the mentioned patch above. Just copy the DebertaTokenizer class to your runtime.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
T5 Base length of Tokenizer not equal config vocab_size 
Author: ari9dam
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: Installed from git

## Issue
The `len(AutoTokenizer.from_pretrained("t5-base"))` is `32100` but the `T5ForConditionalGeneration.from_pretrained("t5-base").config.vocab_size`  is `32128`. Seems to be a similar issue to that of : https://github.com/huggingface/transformers/issues/2020



Comments:
duplicate of https://github.com/huggingface/transformers/issues/4875 I think

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
context manager for seeding, or generating fixed random tensor.
Author: sadakmed
Pull request: no

# 🚀 Feature request

context manager for torch random seed, where the seed is fixed only inside 


## Motivation

in some integration test, an input required is very large to be hardcoded, and the ids_tensor provide only int32 examples.
However to fix this input either we can use NumPy seed, but probably it will compromise the randomness of other parts of the test.
I think a context manager where the seed is fixed only inside, it would benefit alot.

example of issues needing large fixed tensors as input #9951 #9954 


if there's any alternative please suggest?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.