noamaanMulla-03/transformers-issues huggingface dataset:
correct incorrect dimension comment in Longformer model
Author: fredo838
Pull request: yes

This PR fixes a comment that incorrectly states the dimensions of a certain tensor in the `Longformer` model, confusing any reader trying to understand the code. The comment for the corresponding `TFLongformerX` is correct. 

- [x ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

Tagging @patrickvonplaten  (Longformer)



Comments:
`sp_model_kwargs` param missing at unpickle in `XLMRobertaTokenizer`
Author: PhilipMay
Pull request: no

When `XLMRobertaTokenizer` is unpickled the `sp_model_kwargs` is not set. See:

https://github.com/huggingface/transformers/blob/35cd8eed887891bee60194a95adc35b884f68f55/src/transformers/models/xlm_roberta/tokenization_xlm_roberta.py#L178

PS: I will provide a fix.

Comments:
fix pr at #11430

Closed by #11430, thanks @PhilipMay!
RoBERTa adds two sep tokens
Author: david-waterworth
Pull request: no

## Environment info

- `transformers` version: 4.5.1
- Platform: Linux-5.4.0-72-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.6
- PyTorch version (GPU?): NA
- Tensorflow version (GPU?): NA
- Using GPU in script?: NA
- Using distributed or parallel set-up in script?: NA


## Information

I'm using RoBERTa. I noticed when I pass a text pair, two <eos> tokens are added - see the linked code

https://github.com/huggingface/transformers/blob/f37f2adb68b186f175a81a870cc526349385b9a8/src/transformers/models/roberta/tokenization_roberta_fast.py#L230

This differs to the BERT implementation

https://github.com/huggingface/transformers/blob/60d5bda4fd0381075a300dc11903c76df694bd1c/src/transformers/models/bert/tokenization_bert_fast.py#L255

Is this intentional? I'm trying to create a hybrid BERT/RoBERTa style training strategy. I want to pass two sentances but I don't want to use NSP so I was hoping to use my existing custom RoBERTa tokenizer. I ran into two issues - this, and the fact that as per the comment "RoBERTa does not make use of token type ids, therefore a list of zeros is returned." - this comment also doesn't appear correct, from what I can see Roberta's Tokenizer simply does not return `token_type_ids` I've not figured out why yet 

EDIT: seems the default for `return_token_type_ids` is different for BERT (true) and RoBERTa (false).

Also as far as I can see RoBETa will use `token_type_ids` if they're provided, It's just that the tokeniser has been coded to return all zeros.

https://github.com/huggingface/transformers/blob/f37f2adb68b186f175a81a870cc526349385b9a8/src/transformers/models/roberta/modeling_roberta.py#L79 



Comments:
> Is this intentional? 

Yes. It is in line with the original implementation. (Check [link](https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.md) for more information).

> I ran into two issues - this, and the fact that as per the comment "RoBERTa does not make use of token type ids, therefore a list of zeros is returned." - this comment also doesn't appear correct, from what I can see Roberta's Tokenizer simply does not return token_type_ids I've not figured out why yet

The comment is also correct. The original RoBERTa doesn't even have a token_type layer and the huggingface Roberta has one which is just full of zeros (i.e. does nothing as long as you don't do something with it) and only exists due to legacy reasons (check #2871).



Thanks, it appears I can put a BERT tokeniser in front of a RoBERTa mlm model and get why I want.

You can put every tokenizer in front of RoBERTa, but when you use the pre-trained weights you should stick to the original one as it will otherwise lead to garbage. :)


Yeah I understand that but I'm training from scratch.
DeBERTa Fast Tokenizer
Author: brandenchan
Pull request: no
Labels: name: Good First Issue

name: Good Second Issue

Hi, I am interested in using the DeBERTa model that was recently implemented here and incorporating it into [FARM](https://github.com/deepset-ai/FARM) so that it can also be used in open-domain QA settings through [Haystack](https://github.com/deepset-ai/haystack).

Just wondering why there's only a Slow Tokenizer implemented for DeBERTa and wondering if there are plans to create the Fast Tokenizer too. Thanks in advance!

Hi @stefan-it! Wondering if you might have any insight on this?  

Comments:
Hi @brandenchan ,

I think it should be easier with version 2 of DeBERTa, because they use a "normal" sentence piece model now:

https://github.com/huggingface/transformers/pull/10018

So having a fast alternative would be great. 

(The new 128k vocab size should really boost performance on QA tasks!)



Indeed, this would be a very nice addition and way easier to implement than for the first DeBERTa. I'm adding the `Good Second Issue` label so that a community member may work on it. @brandenchan or @stefan-it feel free to take it too if you feel like it!

Hi, I am looking for my first open source contribution. May I take this if its still available?

Yes, of course! Thank you!

@ShubhamSanghvi Maybe wait until #10703 is merged.

Hi, as far as I understand I will have to add tokenizer files for debarta_v2 to implement the fast tokenizer? 

May I know how could I get the tokenizer files for deberta_v2 models and how to upload them to the intended destinations,  which I believe should be (for deberta-v2-xlarge) :

https://huggingface.co/microsoft/deberta-v2-xlarge/resolve/main/

Thanks, Shubham

@ShubhamSanghvi Do you only want to implement the fast tokenizer for DebertaV2 or also for Deberta? 

> May I know how could I get the tokenizer files for deberta_v2 models  

I  think this is what you have to figure out. I would check the other models that have a slow sentencepiece tokenizer.

> how to upload them to the intended destinations, which I believe should be (for deberta-v2-xlarge)  

You can not upload them there. Upload them to some kind of a public cloud and request an upload.






@ShubhamSanghvi  Are you planning to create a PR for this issue soon?


Hi @mansimane, I am currently working on it. I am hoping to get it done by next week.
BART summarization, tokenizer not working
Author: shizhediao
Pull request: no

@patil-suraj 
When I am running pytorch/summarization, the logs are as below:
```
Adding AddedToken(content='
', single_word=False, lstrip=False, rstrip=False, normalized=True) to the vocabulary
Adding AddedToken(content='
', single_word=False, lstrip=False, rstrip=False, normalized=True) to the vocabulary
Adding AddedToken(content='<pad>', single_word=False, lstrip=False, rstrip=False, normalized=True) to the vocabulary
Adding AddedToken(content='<mask>', single_word=False, lstrip=True, rstrip=False, normalized=True) to the vocabulary
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
https://huggingface.co/facebook/bart-large-cnn/resolve/main/pytorch_model.bin not found in cache or force_download set to True, downloading to /home/sdiaoaa/.cache/huggingface/transformers/tmpama_iuh8
```
But the embedding did not fit for the added tokens.
The embedding is still [50264] but the token_ides ranges from [0, 50269]

Comments:
Did you resize the embeddings after adding the new tokens ? To resize the embedding 
```python3
model.resize_token_embeddings(len(tokenizer))
```

> Did you resize the embeddings after adding the new tokens ? To resize the embedding
> 
> ```python
> model.resize_token_embeddings(len(tokenizer))
> ```

Thanks so much for your reply!
The issue has been solved by adding that line.
Just curious, is it a common practice to add 'resize_token_emb' function?  because previously, I have not seen this happens and was wondering why it is not included in the official run_summarization code.

Thanks!

Yes, the embeddings need to be resized after adding new tokens.

And yes, you are right, the embedding should be resized in the example.
Improve task summary docs
Author: hamelsmu
Pull request: yes

This PR makes various improvements to the [Summary of Tasks](file:///Users/hamelsmu/github/transformers/docs/_build/html/task_summary.html#named-entity-recognition) docs.  

Instead of providing a summary of changes at the top, I added a comment to all my changes below to give more context behind why I suggested the change.  

@sgugger 

Comments:
heh not sure why CI failed, just see this error message

> Received "killed" signal

@sgugger sorry for the double comment (I commented on an old commit by accident), what I meant to say I made the changes you suggested, LMK if this does a good job of conveying the message!

Thanks again!
transformers import error
Author: KYUNGGUK-CHOI
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.2
- Platform: MACOS
- Python version: Python 3.6.13
- PyTorch version (GPU?): 1.8.0 (cpu)
- Tensorflow version (GPU?): tensorflow-cpu  2.4.1
- Using GPU in script?: no
- Using distributed or parallel set-up in script?:no

### Who can help

@LysandreJik 

I setup my conda envs as below
 
<img width="470" alt="스크린샷 2021-03-23 오후 8 57 52" src="https://user-images.githubusercontent.com/55866896/112142854-70890b80-8c1a-11eb-8752-14857d4529e7.png">

and there are all librarys needed list in !pip3 list 

<img width="310" alt="스크린샷 2021-03-23 오후 9 05 00" src="https://user-images.githubusercontent.com/55866896/112143640-84813d00-8c1b-11eb-8e6c-735d0a57e1c2.png">

but whenever I try to import BertTokenizer(from transformers import BertTokenizer), ImportError occurs.

<img width="662" alt="스크린샷 2021-03-23 오후 8 59 13" src="https://user-images.githubusercontent.com/55866896/112142993-a0381380-8c1a-11eb-8ec1-722d71a947ed.png">

I tried all process in google colab , it works well 
I have no idea why it does not work in my Macbook local environment

please help me

Comments:
Hello! I would guess something is wrongly setup in your local environment; especially it it works in colab! Are you sure you're using the `python` from the same environment as your `pip3`?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Small inconsistency in tokenization_utils for special tokens retrieval
Author: aitor-garcia-p
Pull request: no

Hi there,

This is just a minor issue I have spotted.
When a special token (cls_token, sep_token, etc.) is accessed in an instantiated Tokenizer, this is the piece of code executed to retrieve the property:

```python
    @property
    def eos_token(self) -> str:
        """
        :obj:`str`: End of sentence token. Log an error if used while not having been set.
        """
        if self._eos_token is None and self.verbose:
            logger.error("Using eos_token, but it is not set yet.")
            return None
        return str(self._eos_token)
```

The None check is tied to the verbose flag, so when the verbose flag is set to False the condition is not triggered, and even if the special token is None, a 'None' literal string is returned (the last line).
It happens the same for all the special tokens, leading to unexpected behavior if you are expecting an actual None outside the tokenizer. I think that the "verbose" flag and the "is None" should be checked in separated conditionals.

The mentioned code can be located at:
https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils_base.py#L949

Thank you very much.
 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
ReformerEmbedding unclear behavior
Author: fostiropoulos
Pull request: no

## Environment info
- `transformers` version: 4.3.3
- Platform: Linux-4.15.0-136-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help
@patrickvonplaten
## Information

Model I am using (Bert, XLNet ...): Reformer

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: MNIST

## To reproduce

Steps to reproduce the behavior:

1. Leave max_position_embeddings undefined or different than axial_pos_embedding shape
2. Observe the assert get triggered for large sequence lengths (within limit of axial_pos_embedding) https://github.com/huggingface/transformers/blob/master/src/transformers/models/reformer/modeling_reformer.py#L255

## Expected behavior

* No assert

## Additional Details:

`max_position_embeddings` is only used by `PositionEmbeddings`, thus if we provide `axial_pos_embds`  in the configuration, the `max_position_embeddings` will be not used for positional embedding. Moreover it is considered for factorizing `num_buckets` in `LSHSelfAttention` layer. 

Thus in a scenario of using axial positional embedding, both the assert check will be useless as well as the [bucket factorization](https://github.com/huggingface/transformers/blob/master/src/transformers/models/reformer/modeling_reformer.py#L711)

Comments:
I think I don't fully agree here...`max_position_embeddings` is important even if `axial_pos_embedding` is used. If a user makes use of `axial_pos_embeddings` then `max_position_embeddings` is clearly defined by the tuple of `axial_pos_embeddings`. IMO, it's important that the user fully understands how `axial_pos_embeddigs` work and should therefore also set `max_position_embeddigns`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[debug utils] activation/weights underflow/overflow detector
Author: stas00
Pull request: yes

This PR came to be out of the overflow issue we have been dealing with in t5/mt5/gpt-neo due to bf16 pretrained models. This PR:

* adds a new file `debug_utils.py`
* adds new helper debug class `DebugUnderOverflow` and function `detect_overflow` for doing the same for any tensor variable (useful for detailed debug).
* extends `Trainer` to support `--debug underflow_overflow` which automatically activates this detector - no changes to the code required
* overloads the old `--debug` which for some reason was used for very specific tpu debug prints yet, so folding that feature into now multi-optional `--debug` (similar to `--sharded_ddp`). the old unqualified `--debug` is now specific `--debug tpu_metrics_debug`. I know it sort of breaks back-compat for `--debug`, but since it's debug it's hopefully OK. 
* creates a new doc `debugging.rst` - will add some more useful debug recipes into it later

I'm open to suggestions of different namings to all of the new things...

@LysandreJik, @sgugger 

Comments:
> I haven't commented on each print statement, but they should use the logger maybe? 

I started with it first and then replaced with print, because we need all the horizontal space and the really busy long pre-amble is just getting in the way, IMHO. I also don't see what useful information it'd contribute because the tool raises an exception when it detects the problem. Finally, what if someone disabled the logger - it'd not be able to do its work then. Please correct me if I'm missing something.

> Also I think the `DebugActivationOverflow` should be documented in our internals API doc (since we tell people to use it in their own Trainers). `internal/trainer_utils` is probably the place for that.

Will do. Thank you for suggesting where to put it.

Converted to .rst - found this new tool https://github.com/miyakogi/m2r that did it well, just needed to clean up a weird quirk.


All links have been added and tweaked the doc some more to improve readability.

OK, the original functionality has been expanded to include a lot more useful information. Please see the updated copious documentation both in the docstring and the user docs.

The main changes are that:
1. we now print each frame separately and include inputs/outputs/weights
2. there is a tracing mode which can easily trace any number of batches at will

I wasn't sure how I could integrate the new features into the limited `--debug underflow_overflow` interface as it now has 3 optional parameters. So for now these can be activated directly from the script. If you can think how I could make these work with ``TrainingArguments`` I'm all ears.

As this has changed a lot inside I'd appreciate another look from @sgugger and @LysandreJik - no rush please. And thank you!
[DeepSpeed] fp32 support
Author: stas00
Pull request: yes
Labels: name: DeepSpeed

Things we need to sync with the upcoming `deepspeed==0.3.16` release:

- `zero.Init` now takes a config as an argument
- fp32-support integration, plus doc and tests
- start troubleshooting section

### Future TODO

will probably do in the next PR:
- switch `from_config()` to perform the same `zero.Init` as `from_pretrained` + add test.

###  Blocking events

PRs waiting to be merged before this PR can be merged:

- [x] https://github.com/microsoft/DeepSpeed/pull/1008 `zero.Init(config=ds_config)` new arg 
- [x] https://github.com/microsoft/DeepSpeed/pull/1004 fp32 support
- [x] new release is needed 0.3.16

@sgugger 

Comments:
That's correct. 

Earlier I tried to use `cur_version>pre_version` so it'd already work with master version, but then people were reporting bugs because they were on some older master version, so while this is less convenient, it avoids invalid bug reports and saves time to all ;)
transformers.models.auto.tokenization_auto
Author: Sankalp1233
Pull request: no

I installed the version transformers 3.5.1 to get the version in GitHub using !pip3 install transformers==3.5.1 and !pip3 install transformers but then when I try to install SentenceTransofrmer using : from sentence_transformers import SentenceTransformer I get ModuleNotFoundError: No module named 'transformers.models.auto.tokenization_auto'. I am not sure how to resolve this issue.

Comments:
Could you reproduce this in a colab so that we can take a look? Thanks!

I reworked my code so I didn't get the error anymore I'm honestly not sure how it got fixed

I think it maybe because I didn't restart the runtime


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
not created config.json in Wav2Vec2ForCTC for ASR
Author: Kowsher
Pull request: no

After saving the trained file I can to read the model. Here gives the error 
/content/gdrive/MyDrive/wav2vec2-large-xlsr-hindi' is the correct path to a directory containing a config.json file

Here is my reading model
model = Wav2Vec2ForCTC.from_pretrained("/content/gdrive/MyDrive/wav2vec2-large-xlsr-hindi").to("cuda")


from transformers import TrainingArguments

training_args = TrainingArguments(
  output_dir="/content/gdrive/MyDrive/wav2vec2-large-xlsr-hindi",
  group_by_length=True,
  per_device_train_batch_size=16,
  gradient_accumulation_steps=2,
  evaluation_strategy="steps",
  num_train_epochs=30,
  fp16=True,
  save_steps=400, #this would mean every 400 steps model gets saved which also means Google drive gets full
  eval_steps=400,
  logging_steps=400,
  #learning_rate=3e-4,
  learning_rate=0.1, # this is just for demo
  warmup_steps=500,
  save_total_limit=2,
)

In my saved model there is not created config.json 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
some bugs about mbart50 for spanish
Author: songwangnlp
Pull request: no

```
from transformers import MBartForConditionalGeneration, MBart50TokenizerFast

model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")
tokenizer = MBart50TokenizerFast.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")

article_en = "Hello World"
encoded_en = tokenizer(article_en, return_tensors="pt", padding='longest', truncation=True, max_length=1024)
generated_tokens = model.generate(**encoded_en, forced_bos_token_id=tokenizer.lang_code_to_id["es_XX"])
#tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
text_es = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True, clean_up_tokenization_spaces=True)
```
text_es
['El Presidente (habla en inglés): Doy las gracias al representante de la República Islámica del Irán por su declaración.']

Looks like this model definitely has some bugs for spanish since this is an easy translation task. the correct answer is "Hola Mundo"

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Adds use_auth_token with pipelines
Author: philschmid
Pull request: yes

# What does this PR do?

This PR adds `use_auth_token` as a named parameter to the `pipeline`. Also fixed `AutoConfig.from_pretrained` adding the `model_kwargs` as `**kwargs` to load private model with `use_auth_token`. 

**Possible Usage for `pipeline` with `use_auth_token`:**
with model_kwargs
```python
hf_pipeline = pipeline('sentiment-analysis',
                       model='philschmid/sagemaker-getting-started',
                       tokenizer='philschmid/sagemaker-getting-started',
                       model_kwargs={"use_auth_token": "xxx"})
```
as named paramter
```python
hf_pipeline = pipeline('sentiment-analysis',
                       model='philschmid/sagemaker-getting-started',
                       tokenizer='philschmid/sagemaker-getting-started',
                       use_auth_token = "xxx")
```

cc @Narsil 

Comments: