noamaanMulla-03/transformers-issues huggingface dataset:
actions CI self-scheduled: run_examples torch even if run_torch_tests fails
Author: sshleifer
Pull request: no
Labels: name: wontfix


![image](https://user-images.githubusercontent.com/6045025/89939113-7f6c0200-dbe5-11ea-984d-54c2ff749daa.png)



Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Corrected typo: maked → masked
Author: miguelvictor
Pull request: yes

# What does this PR do?

Fixed small typo in the BERT documentation.

## Before submitting
✅ This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

@LysandreJik @sgugger

Comments:
Ugh and I copied that docstrings to all other models... Thanks for fixing!
Deleting position IDS when fine-tuning BERT
Author: Bushra-Aljbawi
Pull request: no
Labels: name: wontfix

Hi, 

I want help regarding this problem: 

I want to fine-tune BERT model on my custom dataset to fill in the missing words. So, I'm using run_language_modeling to do this. However, as my dataset contains sets of words and not real sentences, I don't really care about the order so I want to try deleting the position encoding by setting it to "zero tensor" to not affect the model. 

I would appreciate any help on how to add my costume position_IDS to run_language_modeling parameters? If it not possible to do so, is there any other way to fine-tune BERT with costume position_IDS? 

Thanks!  

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
2 Deberta test failures
Author: sshleifer
Pull request: no

I suspect these are related to recent tokenizer changes:

https://github.com/huggingface/transformers/runs/1236753957?check_suite_focus=true

```
FAILED tests/test_tokenization_deberta.py::DebertaTokenizationTest::test_torch_encode_plus_sent_to_model
FAILED tests/test_modeling_deberta.py::DebertaModelIntegrationTest::test_inference_classification_head
```

Comments:
Duplicate of https://github.com/huggingface/transformers/issues/7565

Working on it in https://github.com/huggingface/transformers/pull/7645
does tokenizer support emoji?
Author: steveguang
Pull request: no

Hi, I have the code below and it always encodes emoji as "unk". Can someone tell me what I should do? Thanks

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)
s =" 😃 hello how are you"
tokenizer.tokenize(s)
['[UNK]', 'hello', 'how', 'are', 'you']

Comments:
Hi! The tokenizer you're using (`bert-base-uncased`) was not trained with emojis, therefore it cannot tokenize them correctly. You should add this token to the tokenizer vocabulary:
```py
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)
s =" 😃 hello how are you"

tokenizer.add_tokens("😃")
print(tokenizer.tokenize(s))

# ['😃', 'hello', 'how', 'are', 'you']
```

Please be aware that the model you're using should have its embedding matrix updated to include the embedding for the new token added. You can see the [documentation here](https://huggingface.co/transformers/internal/tokenization_utils.html?highlight=add_token#transformers.tokenization_utils_base.SpecialTokensMixin.add_tokens), here's how you should update your model embedding matrix:
```py
# Let's see how to increase the vocabulary of Bert model and tokenizer
tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')

num_added_toks = tokenizer.add_tokens(['new_tok1', 'my_new-tok2'])
print('We have added', num_added_toks, 'tokens')
 # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e., the length of the tokenizer.
model.resize_token_embeddings(len(tokenizer))
```

Thanks, @LysandreJik !  I have another question. When I train using tweets, since there is a lot of noise, a tweet like 'This is soooo good' would be a problem for BERT tokenizer cuz "soooo" is not in the vocabulary. Is there a method to add all of them? Right now I am thinking about a kinda ugly way, just use nltk tweettokenizer to process all tweets and add to vocab with words, emoji, etc that appear frequently

Hi @steveguang, sentences like `This is soooo good` actually won't be a problem for the BERT tokenizer, as it can decompose the word `soooo` in multiple tokens:

```py
>>> from transformers import BertTokenizer
>>> tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
>>> tokenizer.tokenize("This is soooo good")
['This', 'is', 'so', '##oo', '##o', 'good']
```

However, when working with a dataset that seems to have a lot of unknown tokens, it is generally a good idea to identify the tokens that come up relatively often and to add them to your tokenizer. A good example would be the emojis mentioned above, as these are an important attribute to the meaning of the sentence.
fixed typo in warning line 207.
Author: Berowne
Pull request: yes

replace 'men_len' with 'mem_len' to match parameter name

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), 
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? 


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

Comments:
Fix check for xla in PreTrainedModel.save_pretrained()
Author: fteufel
Pull request: yes



# What does this PR do?

Added is_torch_tpu_available() to the condition for saving a model as xla model when calling `PreTrainedModel.save_pretrained()`
The `xla_device` property of `config` can also be `True` on a non-xla device, when loading a checkpoint that was previously trained and saved on xla.

Loading a model that was trained on xla was fixed previously with #5636 , this PR fixes the problem of saving such a model again.

Fixes #7695



## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), 
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? 


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.


Comments:
save_pretrained() does not check if xla is available
Author: fteufel
Pull request: no

## Environment info

- `transformers` version: 3.3.1
- Platform: Linux
- Python version: 3.6
- PyTorch version (GPU?): 1.4
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

## Information

Model I am using (Bert, XLNet ...): Bert


The problem arises when using:
* [ X] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. load any model trained on TPU with `BertModel.from_pretrained(tpu_checkpoint_path)`
2. run/train the model - works fine
3. save the model with `model.save_pretrained(save_path)`

```
line 720, in save_pretrained
    import torch_xla.core.xla_model as xm
ModuleNotFoundError: No module named 'torch_xla'
```
## Expected behavior

I am pretraining a LM on TPU, and for the downstream task fine-tuning I load the saved checkpoints on a non-TPU device.
Loading works fine now (#5636), but saving again does not.

`save_pretrained` should check whether the device is still xla - the original config attribute that is used in `save_pretrained` to check for the device persists when loading the xla model on another device:
```
getattr(config, 'xla_device')
True
```
It is easy to fix by changing the config attribute `setattr(config, 'xla_device', False)` in the script, but I would still consider it a bug.


Comments:
Reason behind the layers taken for distilbert-multilingual
Author: divyag11
Pull request: no

Hi,
I went across the training code for distilbert. I can see that the distillation process is used to train the distilbert model from bert model.

What is the reason only the **layers [0, 2, 4, 7, 9, 11]** have been taken to train distilbert model?Is there any idea about the layers behind this?

And,what is the **last two layers in distilbert layer** corresponds and are they equivalently similar to the **9 and 11** layer of the original bert model?

This was the training code i referred to:
https://github.com/huggingface/transformers/blob/master/examples/distillation/scripts/extract_distilbert.py

Comments:
With distillation, the initialization is really important in order to obtain good results. You can read the [paper](https://arxiv.org/pdf/1910.01108.pdf) to have more information, look for section 3 and "Student Initialization"!

The important part is the following:

_Student initialization In addition to the previously described optimization and architectural choices,
an important element in our training procedure is to find the right initialization for the sub-network to
converge. Taking advantage of the common dimensionality between teacher and student networks,
we initialize the student from the teacher by taking one layer out of two._

Why not [0, 2, 4, 6, 8, 10]?
Trainer callback breaks old code
Author: changjonathanc
Pull request: no

https://github.com/huggingface/transformers/blob/ba4bbd92bcb55febbfa06aaa1551738388ec7eb0/src/transformers/trainer_callback.py#L438-L447

Currently it depends on the fact that evaluate() will first call `self.prediction_loop` https://github.com/huggingface/transformers/blob/ba4bbd92bcb55febbfa06aaa1551738388ec7eb0/src/transformers/trainer.py#L1181

which will then call `.callback_handler.on_prediction_step `
https://github.com/huggingface/transformers/blob/ba4bbd92bcb55febbfa06aaa1551738388ec7eb0/src/transformers/trainer.py#L1270

But in my old code (3.1.0), I subclass Trainer and overwrite evaluate(), without calling self.prediction_loop.

And results in this error:
```
self.prediction_bar.close() 
AttributeError: 'NoneType' object has no attribute 'close'
```

I propose we add `on_predict_begin`  and `on_predict_end`.

Comments:
Or add this check in `on_evaluate`
```python
if self.prediction_bar is not None:
    self.prediction_bar.close()
```


I think the second solution is better (since it should be there in any case). We can add the events you suggest in the first one if we find other use cases that need them.

Do you want to tackle this in a PR?

I was thinking we can move the opening and closing of `prediction_bar` into separate events, so we don't need the `if` statements. 
But it can be hard to debug if someone miss the opening, so it's probably unnecessary.

I will create a PR with the second solution.
Cannot use the RobertaForMultipleChoice model for processing multiple choice questions with 4 options
Author: h56cho
Pull request: no
Labels: name: wontfix

Hello,

When I try to use the `RobertaForMultipleChoice` pretrained model with the code below, it generates an error:

```python
from transformers import RobertaTokenizer, RobertaForMultipleChoice
import torch
tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
model = RobertaForMultipleChoice.from_pretrained('roberta-base')
prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
choice0 = "It is eaten with a fork and a knife."
choice1 = "It is eaten while held in the hand."
choice2 = "It is eathen with a napkin."
choice3 = "It is not eatable."
labels = torch.tensor(0).unsqueeze(0)  # choice0 is correct (according to Wikipedia ;)), batch size 1
encoding = tokenizer([[prompt, prompt,prompt,prompt], [choice0, choice1, choice2, choice3]], return_tensors='pt', return_token_type_ids=True,padding=True)
```

The error message is:

```python

  File "/Users/hyunjindominiquecho/opt/anaconda3/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 534, in _batch_encode_plus
    ids, pair_ids = ids_or_pair_ids

ValueError: too many values to unpack (expected 2)
```

What am I doing wrong here? Thank you.



Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Good question. `xxxForMultipleChoice` models are actually a bit tricky. The way you should provide the data to the tokenizer is as follows:

```
prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
choice0 = "It is eaten with a fork and a knife."
choice1 = "It is eaten while held in the hand."
choice2 = "It is eathen with a napkin."
choice3 = "It is not eatable."

encoded_dict = bert_tokenizer([prompt, prompt, prompt, prompt], 
                                 [choice0, choice1, choice2, choice3], 
                                return_tensors='pt', 
                                padding='max_length')
```
Note the difference: we provide 2 lists to the tokenizer rather than one. The first list contains the first sequence of every training example, the second list contains the second sequence. Every training example will then be encoded as `[CLS] prompt [SEP] choice x [SEP]`.

For more details, see [here](https://github.com/huggingface/transformers/issues/7701#issuecomment-707149546).
namespace object has no attribute to "enc_only"
Author: zy329jy
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help
when I running the distillation.py then 
 File "E:/transformers-master/examples/seq2seq/distillation.py", line 370, in create_module
    elif args.enc_only:
AttributeError: 'Namespace' object has no attribute 'enc_only'
 how can I deal  with this problem??  thx a lot.

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
If you do not want to use encoder only, I think it is fine to just comment that elif clause out

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Fine Tuning SciBERT NLI model
Author: duttaprat
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Hi,
I am a novice in the domain of fine-tuning any Transformer models using my own dataset. I want to fine-tune the SciBERT NLI model(https://huggingface.co/gsarti/scibert-nli) using my dataset. The dataset formation is
S1 [SEP] S2 [SEP] Inference

I am not sure how to fine-tune my dataset on the SciBERT NLI model(https://huggingface.co/gsarti/scibert-nli).

I am sorry for asking a very simple question. But any suggestion or link helps me. Thanks in advance. -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


Hi,
I am a novice in the domain of fine-tuning any Transformer models using my own dataset. I want to fine-tune the SciBERT NLI model(https://huggingface.co/gsarti/scibert-nli) using my dataset. The dataset formation is
S1 [SEP] S2 [SEP] Inference

I am not sure how to fine-tune my dataset on the SciBERT NLI model(https://huggingface.co/gsarti/scibert-nli).

I am sorry for asking a very simple question. But any suggestion or link helps me. Thanks in advance. 

Comments:
Take a look at the official tutorial on fine-tuning a model on your own dataset here: https://huggingface.co/transformers/custom_datasets.html

What's your dataset about? Is it text classification, question answering?

BTW, please post any questions which are not bugs/new features you would like to see added on the [forum](https://discuss.huggingface.co/) rather than here. 

Thanks @NielsRogge for the reply. 

My dataset contains two clinical sentences (S1, S2) and its corresponding relation ('Entailment'/'Contradiction'/'Neutral'). So it basically contains three columns.  

Ok so that's sentence pair classification. There's an example notebook on that [here](https://github.com/NadirEM/nlp-notebooks/blob/master/Fine_tune_ALBERT_sentence_pair_classification.ipynb).

Oh that's really helpful. Thank you again @NielsRogge .
[marian] Automate Tatoeba-Challenge conversion
Author: sshleifer
Pull request: yes
Labels: name: marian

This allows conversion of marian models from the Tatoeba-Challenge repo through the command line,
with instructions at `scripts/tatoeba/README.md`. This was previously impossible.
Tests are in `examples/` because the conversion requires examples dependencies (wget and pandas).

In my opinion, it would be a lot of work for very little benefit to remove these dependencies.

The goal of this PR is to allow @jorgtied to upload his own models.

Comments:
Avoid unnecessary DDP synchronization when gradient_accumulation_steps > 1
Author: noamwies
Pull request: yes

# What does this PR do?


This PR avoid unnecessary ```DistributedDataParallel``` synchronization when gradient_accumulation_steps > 1 by using  ```DistributedDataParallel.no_sync```.


This lead to speedup when training with multiple gpu's for example the ```run_language_modeling.py``` complete wiki-2 epoch in 85 seconds instead of 111

```bash
python run_language_modeling.py --output_dir=runs --model_type=gpt2 --model_name_or_path=gpt2 --per_device_train_batch_size 6 --do_train --train_data_file=$TRAIN_FILE  --gradient_accumulation_steps=32 --fp16 --block_size 513 --overwrite_output_dir
```


## Before submitting
- [x]  Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), 
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? 


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@patrickvonplaten

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

Comments: