noamaanMulla-03/transformers-issues huggingface dataset:
[examples run_clm] fix _LazyModule hasher error
Author: stas00
Pull request: yes

This PR fixes a problem I introduced in https://github.com/huggingface/transformers/pull/11145 and reported in https://github.com/huggingface/transformers/issues/11166 

`datasets.fingerprint.Hasher` fails to run 

```
hasher = Hasher()
hasher.update(tokenize_function)
```
getting:
```
TypeError: cannot pickle '_LazyModule' object
```

Because the logger object contains a lazy import.

The error was subtle as the exception was caught and not propagated but instead a warning was logged, which I didn't notice in the first place. Warnings aren't a great way to communicate problems. So we were getting now:

> [WARNING|tokenization_utils_base.py:3144] 2021-04-09 09:46:31,368 >> Token indices sequence length is longer than the specified maximum sequence length for this model (1462828 > 1024). Running this sequence through the model will result in indexing errors
> [WARNING|run_clm.py:326] 2021-04-09 09:46:31,368 >> ^^^^^^^^^^^^^^^^ Please ignore the warning above - this long input will be chunked into smaller bits before being passed to the model.
> 04/09/2021 09:46:31 - WARNING - datasets.fingerprint -   Parameter 'function'=<function main.<locals>.tokenize_function at 0x7f434d90da60> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.

So I fixed this by moving the logger object fetching to outside of the function to be hashed and then it all works.

Fixes: https://github.com/huggingface/transformers/issues/11166 

@sgugger 


Comments:
added json dump and extraction of train run time
Author: philschmid
Pull request: yes

# What does this PR do?

This PR adjusts to the latest metric exposing changes that `train_runtime` is now logged as `hh:mm:ss.ms`. So instead of extracting the `train_runtime` from the logs it is using the `sagemaker-sdk` to get the full train time. Additionally, I added a JSON dump for all tests to share the result easier, when opening a new PR to upgrade the HF DLC. 

Comments:
LM finetuning on domain specific unlabelled data
Author: sML-90
Pull request: no

Hello Team,
Thanks a lot for the awesome work! 
Can you please tell me how to finetune a(any) MLM model on domain specific corpus ? I am following this [link](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/language_modeling.ipynb) obtained from the huggingface documentation. Is this the procedure I should be following ? if this is how it is done, how will this update the vocabulary to adapt to new tokens of my domain specific corpus ?

Thanks in advance. 

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!
Getting value of [UNK] labels
Author: Javier-Jimenez99
Pull request: no
Labels: name: wontfix

name: Core: Tokenization

name: Ex: Named Entity Recognition

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

I have created a NER model based ob Bert with this library, but I have a problem when  I run my model due to `[UNK]`. Sometimos there are entities that aren't on my vocab so they are marked as unkowns so I cant know what they are. 

I know I can't revert `[UNK]` label, so would like to be able to define the words that would be unknown before the processing of the sentence.          

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:  [https://stackoverflow.com/questions/60192523/get-the-value-of-unk-in-bert](url)

Comments:
Did you try using the `add_tokens` method on the tokenizer alongside the `resize_token_embeddings` on the model, to add your tokens to the vocabulary? The won't be marked as `[UNK]` this way, but will instead receive brand new embeddings (which need to be trained).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@LysandreJik  Hello, I have the same problem。
I want to implement [pointer-generator](https://arxiv.org/abs/1704.04368) by BertTokenizer. Pointer-generator can generate OOV tokens in the inputs by dynamically extending vocab. And there is no way to add all the tokens directly, especially in the test set.
Do you have any good solutions?

Have you tried using the `add_tokens` method on the tokenizer and the `resize_token_embeddings` method on your model?

I tried the dumbest solution:
```
bert_tokens = tokenizer.tokenize(query)
tokens = []
pre_text = ""
for i in range(len(bert_tokens)):
    bert_token = bert_tokens[i].replace("##", "")
    if i+1 < len(bert_tokens):
        post_token = bert_tokens[i+1].replace("##", "")
    else:
        post_token = ""
    if bert_token == '[UNK]':
        token = str(
            re.match(f"{pre_text}(.*){post_token}(.*)",
                     query).group(1))
        tokens.append(token)
        pre_text += token
    else:
        tokens.append(bert_token)
        pre_text += bert_token
return tokens
```
Run mlm pad to multiple for fp16
Author: ak314
Pull request: yes

# What does this PR do?
This PR uses padding to a multiple of 8 in the run_mlm.py language modeling example, when fp16 is used. Since the DataCollatorForLanguageModeling did not initially accept the pad_to_multiple_of option, that functionality was added.
Fixes #10627 

## Before submitting
- [X] Did you write any new necessary tests?

## Who can review?
@sgugger 

Comments:
AttributeError: 'tensorflow.python.framework.ops.EagerTensor' object has no attribute 'new_ones'
Author: Tortoise17
Pull request: no

I have environment with following

```
torch=1.7.1+cpu
tensorflow=2.2.0
transformers=4.2.2
Python=3.6.12 
```

and I am using below command 

```
input_ids = tokenizer.encode('accident', return_tensors='tf')
greedy_output = model.generate(input_ids, max_length=50)
print("Output:\n" + 100 * '-')
```

but I get the error below


```
  File "/home/anaconda3/envs/myenv/lib/python3.6/site-packages/transformers/generation_utils.py", line 368, in _prepare_attention_mask_for_generation
    return input_ids.new_ones(input_ids.shape)

AttributeError: 'tensorflow.python.framework.ops.EagerTensor' object has no attribute 'new_ones'
```

Earlier, until October, the same was working perfectly. May I ask help which error or conflict I am making./?


Comments:
Did you fix this, and if so, how? I'm having the same problem and nothing's working for me so far.

**EDIT**: Fixed it, I was using `tf` tensors with a (PyTorch) `AutoModel` instead of a `TFAutoModel`.
Add 3D attention mask to T5 model (#9643)
Author: lexhuismans
Pull request: yes

# What does this PR do?
It allows for 3D attention mask in T5 model (modeling_t5.py). 

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #9643 

This  is a solution for allowing the 3D attention mask in the T5 model by making it broadcastable. It is based on what is used in BERT. 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
Hey @lexhuismans,

Thanks a lot for your PR! Could you also add a test to verify that T5 can be used with a 3D mask?


Hey @patrickvonplaten,

Thanks for your message. I had a shot in adding a test for the 3D attention mask. The test passed on my device. 

I based the test on a similar test for a default attention mask in test_modeling_bert.py. (Not sure if bert already tests for 3D attention mask?) 

Also, I did a rebase before pushing which is why there are so many other commits in-between. 

Let me know if something is still missing or incorrect so I can have a look at it. 

I made a new PR with just the two commits #11197.
Loading pretrained mBART model always generate the same output
Author: Skylixia
Pull request: no

Hi,

I trained mBART with pytorch lightning with the model MBartForConditionalGeneration.from_pretrained('facebook/mbart-large-cc25') to do summaries.  As output I got a checkpoint.ckpt. Moreover, I used model.model.save_pretrained to have a config.json and pytorch_model.bin. 
During the training and testing, I saw what the model was generating as summaries and I got satisfying results. 
However, when I load it back into transformers, the output it generates is always the same no matter the input. I can see that this output comes from my training data but during the training and testing the model was not doing this. The model was in fact generating output with a relation to the input which is not the case here as it always outputs the same thing. 
I suppose there must be a mistake on how I load and use the pretrained model but I don't know what. 
This is how I do it: 

```
configuration = MBartConfig.from_json_file("config.json")
model = MBartForConditionalGeneration.from_pretrained("pytorch_model.bin",
config="configuration")
tokenizer = MBartTokenizer.from_pretrained('facebook/mbart-large-cc25')
inputs = tokenizer([ARTICLE_TO_SUMMARIZE], max_length=1024, return_tensors='pt')
summary_ids = model.generate(inputs['input_ids'], num_beams=4, max_length=150, early_stopping=True)
print([tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in summary_ids])
```


Thanks in advance for the help

Comments:
Minor typos fixed
Author: cronoik
Pull request: yes

# What does this PR do?
Fixes minor typos.

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

Documentation: @sgugger


Comments:
Added documentation for data collator.
Author: fghuman
Pull request: yes

# What does this PR do?


This PR aims to improve  coverage of the documentation for the Data Collators. 


Fixes #9035


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?






Comments:
Thanks again for your contribution!
ELECTRA Model
Author: josecannete
Pull request: no

## 🚀 Feature

New Transformer based model: ELECTRA

## Motivation

Hi guys, did you see the following paper: https://openreview.net/forum?id=r1xMH1BtvB ? There is a new Transformer based model called ELECTRA that seems very interesting and promising. It would be very useful to have a implementation of the model in PyTorch.

## Additional context

Paper: https://openreview.net/forum?id=r1xMH1BtvB


Comments:
Hi @josecannete 

Thanks for the tip! We are busy building other awesome things at the moment, but feel free to start a PR with a first draft and we will be happy to have a look at it 😄 

And note that it's probably better to wait for the author's original code and pretrained weights.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Any news? Do you know if the code was released?

waiting...

The original code is available from here: https://github.com/google-research/electra

Anyone wants to give it a go? We can help!

We're on it! :hugs: 

@LysandreJik I can help you with evaluating the model on downstream tasks to compare it with the original implementation - I'm currently training an ELECTRA model on GPU, so I'm highly interested in using it with Transformers 😅

@LysandreJik If it helps, I believe ELECTRA weights are drop-in replacements into the BERT codebase except we do not use a pooler layer and just take the final [CLS] hidden state for sentence representations.

waiting...+10086

Since v2.8.0 ELECTRA is in the library :)

@LysandreJik Is pretraining of Electra from scratch support available now?

Using default scripts `run_language_modeling.py`?

Hi, I'm trying to fine tune ELECTRA large discriminator for a downstream classification task. I took the [CLS] at the last hidden state as the sentence representation like some Autoencoding pretrained LM (BERT, RoBERTa,...). Is that right? 
Just because my results are not stable.
How to extract the specific output using the method "encoder_output[0]"
Author: bruceszq
Pull request: no

Dear Transformers Team

Thank you very much for Transformers which provides me solutions for relation extraction problems. I have a question. My input is like " [E1]Jack[/E2] was born in [E2]London[/E2]". I want to only extract the sequence output of "[E1] and "[E2]" with the method of "encoder_output[0]". And then concat the [CLS] output and sequence output of "[E1] and "[E2]". Could you help me solve the problem? This is a question that I took more than a month to thank aout the solution but I am not able to solve it. Thank you very much for your help. Thank you!

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!
Fix typo
Author: tma15
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
Transfer learning on bert
Author: gopalpurama
Pull request: no

We take pretrained bert model and passing our dataset we save model on .bin file and we take predictions.
But how can we retrain a model with new data set on top of generated bin file 
Please help me on this issue 

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!
ProphetNet with AttributeError: module 'torch.utils' has no attribute 'checkpoint'
Author: StevenTang1998
Pull request: no

## Environment info

- `transformers` version: 4.5.0
- Platform: Linux-5.4.0-70-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.6
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@patrickvonplaten

## Information

Model I am using ProphetNet:

The problem arises when using:
my own modified scripts (simplified): 
```python
self.model = ProphetNetForConditionalGeneration.from_pretrained(self.pretrained_model_path, config=self.config)
outputs = self.model(
            input_ids,
            attention_mask=input_att,
            decoder_input_ids=decoder_input_ids,
            decoder_attention_mask=decoder_input_att,
            use_cache=False
        )
```
And rised: 
```
File "/home/ruc/tty/TextBox/textbox/model/Seq2Seq/prophetnet.py", line 89, in forward
    use_cache=False
  File "/home/ruc/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ruc/anaconda3/lib/python3.7/site-packages/transformers/models/prophetnet/modeling_prophetnet.py", line 1841, in forward
    return_dict=return_dict,
  File "/home/ruc/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ruc/anaconda3/lib/python3.7/site-packages/transformers/models/prophetnet/modeling_prophetnet.py", line 1725, in forward
    return_dict=return_dict,
  File "/home/ruc/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ruc/anaconda3/lib/python3.7/site-packages/transformers/models/prophetnet/modeling_prophetnet.py", line 1272, in forward
    layer_outputs = torch.utils.checkpoint.checkpoint(
AttributeError: module 'torch.utils' has no attribute 'checkpoint'
```
I think it is the same problem as [#9617](https://github.com/huggingface/transformers/issues/9617) and [#9919](https://github.com/huggingface/transformers/issues/9919).

Comments:
Add new community notebook - wav2vec2 with GPT
Author: voidful
Pull request: yes

* Update:community.md, new nb add
* feat: notebook of wav2vec xlsr ctc decoding with gpt logit adjustment
* Update: Wav2vec2 CTC decoding with gpt2 adjustment


Comments:
Do you want to take a look @patrickvonplaten?

thanks a lot！！！