noamaanMulla-03/transformers-issues huggingface dataset:
Unexpected behavior encoding token_type_ids in GPT models
Author: StuartMesham
Pull request: no
Labels: name: wontfix

The issues #4922 and #1339 have been closed, but bug is still present in the `GPT2Model` code resulting in unexpected behaviour as described in #4922 .

I am trying to use token_type_ids to annotate different languages being input to a multilingual model. I am using my own tokenization code which outputs token_type_ids indicating the language of the tokens. I have run into unexpected behaviour where the embeddings for the first few tokens in my vocabulary are also being used for my token type embeddings. I am using version 3.0.2.

The issue was marked as closed since the GPT2 Tokenizer was no longer outputting token_type_ids, but the `GPT2Model` class still accepts token_type_ids in its forward method.

# Expected Behavior

Either:

1. addition of a separate `nn.Embedding` matrix for token_type_ids [here](https://github.com/huggingface/transformers/blob/3ae2e86baffc1fea8b8b93695fb5a10941fd63dc/src/transformers/modeling_gpt2.py#L352) (such as [in BERTModel](https://github.com/huggingface/transformers/blob/3ae2e86baffc1fea8b8b93695fb5a10941fd63dc/src/transformers/modeling_bert.py#L153))
2. discourage or throw a warning when `token_type_ids` are passed to GPTModel instances

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Create smaller number of heads in attn without pruning using shared parameters
Author: huu4ontocord
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

Instead of prunning heads or masking heads, create new linear layers with views of the larger linear layers that contains all heads.

## Motivation

For various models (Bert, Distilbert, etc.) I would like to be able to experiment on using separate heads without having to prune the heads. For example, training layers with separate heads. There is a way to masks heads, but the computation is still performed over all of the k, q, v tensors, and then masking is performed. 

Is there a way to create views of the heads that you would otherwise prune, so that the extra computation per unused heads is not performed. My understanding is that the attention computation is one of the more expensive operation, since it is quadratic in the seq_len. 

For example, with DistilBert, I was thinking of replacing, q_lin, v_lin, and k_lin with a shared linear layer that is a view into the original q_lin, k_line, v_lin.  

## Contribution

I am thinking of adapting this code from prune_linear_layer from modeling_utils.py. Do you think this will work? Is there a better way to do this?
``
def share_linear_layer_by_index(layer, index, dim=0):
    """ Create a new linear layer (a model parameters) with shared parameters from index of the old layer
        Return new layer with requires_grad=True.
    """
    index = index.to(layer.weight.device)
    W = layer.weight.index_select(dim, index)
    if layer.bias is not None:
        if dim == 1:
            b = layer.bias
        else:
            b = layer.bias[index]
    new_size = list(layer.weight.size())
    new_size[dim] = len(index)
    new_layer = nn.Linear(new_size[1], new_size[0], bias=layer.bias is not None).to(layer.weight.device)
    new_layer.bias.requires_grad = False
    new_layer.weight = W
    new_layer.weight.requires_grad = True
    if layer.bias is not None:
        new_layer.bias.requires_grad = False
        new_layer.bias = b
        new_layer.bias.requires_grad = True
    return new_layer

``

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Bert transformer issue
Author: skiran252
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->
Token indices sequence length is longer than the specified maximum sequence length for this model (5 > 512). Running this sequence through the model will result in indexing errors
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to convert '.bin' model to '.onnx'
Author: AITutorials
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
I can't reproduce the results of tf-xlm-r-ner-40-lang model 
Author: isl-m
Pull request: no
Labels: name: wontfix

I tired to reproduce the results of tf-xlm-r-ner-40-lang model but there were compatibility issues in the token-classification/run_tf_ner.py
I fixed some of these issue but it's still not running as expected, @jplu could you please share with me the transformers/tf version used to produce the results of tf-xlm-r-ner-40-lang model.

i'm using:
transformers = 3.0.2
tensorflow = 2.3.0

Thanks in advance.

Comments:
Hello!

What is the command line you use to train the model?

Hello @jplu 
that's the command line I tried to use to train the model (you added that command to the model description on huggingface community models)


```
cd examples/ner
python run_tf_ner.py \
--data_dir . \
--labels ./labels.txt \
--model_name_or_path jplu/tf-xlm-roberta-base \
--output_dir model \
--max-seq-length 128 \
--num_train_epochs 2 \
--per_gpu_train_batch_size 16 \
--per_gpu_eval_batch_size 32 \
--do_train \
--do_eval \
--logging_dir logs \
--mode token-classification \
--evaluate_during_training \
--optimizer_name adamw
```

Can you use the last version of the trainer, with th ecommand line I used:

```
python run_tf_ner.py \
--data_dir . \
--labels ./labels.txt \
--model_name_or_path jplu/tf-xlm-roberta-base \
--output_dir model \
--num_train_epochs 8 \
--per_gpu_train_batch_size 32 \
--per_gpu_eval_batch_size 64 \
--do_train \
--do_eval \
--do_predict \
--logging_steps 10 \
--evaluate_during_training \
--save_steps 100 \
--overwrite_output_dir
```

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
unexpected behavior on RoBERTa tokenizer when using additional special tokens 
Author: amirdnc
Pull request: no
Labels: name: wontfix

## Environment info
- `transformers` version: 3.0.2
- Platform: Windows-10-10.0.18362-SP0
- Python version: 3.8.3
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <no
- Using distributed or parallel set-up in script?: no

### Who can help
 tokenizers: @mfuntowicz 

## Information


When trying to tokenize using RoBERTa tokenized and a special token and using add_prefix_space=True, the token following the special token does not get a space. 

## To reproduce

Steps to reproduce the behavior:

1. run the following code

```
tokenizer = RobertaTokenizer.from_pretrained("roberta-base")
tokenizer.add_special_tokens('[d_s]')
print(tokenizer('[d_s] test', add_prefix_space=True))
print((tokenizer('test', add_prefix_space=True))
```

## output
{'input_ids': [0, 1296, 2], 'attention_mask': [1, 1, 1]}
{'input_ids': [0, 50271, 21959, 2], 'attention_mask': [1, 1, 1, 1]}


## Expected behavior
{'input_ids': [0, 1296, 2], 'attention_mask': [1, 1, 1]}
{'input_ids': [0, 50271, 1296, 2], 'attention_mask': [1, 1, 1, 1]}

The second token should not change because of the first one.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Using whole word masking on training LM from scratch
Author: uunal
Pull request: no

# ❓ Questions & Help
## Details
Hello everyone,
I wanted to use _whole-word-masking_ in training LM from scratch. I could not have found how to apply this option using Trainer.
I thought this option should be managed in  "class DataCollatorForLanguageModeling", but I could not find options for _whole-word-masking._
Am I looking at wrong place OR it is not implemented yet? 
If not, is it possible to do with run_language_modeling.py?

**A link to original question on Stack Overflow**:  https://stackoverflow.com/questions/62061578/how-to-use-whole-word-masking-on-training-lm-from-scratch

Any help is appreciated!
Thanks


Comments:
I think it's not implemented yet.

@julien-c any suggestion/thoughts for pretraining with wwm?

NVIDIA/Megatron-LM does wwm on the fly in __ getitem __

We can do something similar in DataCollatorForLanguageModeling or in the dataset 

https://github.com/NVIDIA/Megatron-LM/blob/22c0e300670672e4e0a8604bd6ab89bc28c970a6/megatron/data/bert_dataset.py#L148

Thanks for the suggestion, I'll look into it.

@usuyama The Megatron example is for the BERT dataset which uses wordpiece tokenization. Any suggestions how to do wwm for GPT2 tokenizer?

related #6491

Check if still looking for an answer:
https://github.com/huggingface/transformers/blob/07708793f20ec3a949ccab32cc4fe0c7272dcc4c/src/transformers/data/data_collator.py#L301
Cannot Load roberta tokenizer
Author: Rogerspy
Pull request: no

# ❓ Questions & Help

My `transformers`'s version is `transformers 3.4.0`

I download a Chinese RoBERTa model, where:

```
models
├── RoBERTa_zh_Large_Pytorch
│   ├── config.json
│   ├── pytorch_model.bin
│   └── vocab.txt
```

I want to load the tokenizer from local:

```python
tokenizer_zh = RobertaTokenizer.from_pretrained('./models/RoBERTa_zh_Large_Pytorch/')
```

but I get :

```
OSErrorTraceback (most recent call last)
<ipython-input-4-bdedc9589347> in <module>
----> 1 tokenizer_zh = RobertaTokenizer.from_pretrained('./models/RoBERTa_zh_Large_Pytorch/')

/anaconda3/lib/python3.7/site-packages/transformers/tokenization_utils_base.py in from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
   1647                     ", ".join(s3_models),
   1648                     pretrained_model_name_or_path,
-> 1649                     list(cls.vocab_files_names.values()),
   1650                 )
   1651             )

OSError: Model name './models/RoBERTa_zh_Large_Pytorch/' was not found in tokenizers model name list (roberta-base, roberta-large, roberta-large-mnli, distilroberta-base, roberta-base-openai-detector, roberta-large-openai-detector). We assumed './models/RoBERTa_zh_Large_Pytorch/' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.json', 'merges.txt'] but couldn't find such vocabulary files at this path or url.
```

Comments:
It seems that the tokenizer in that directory is not a Byte level BPE, or at least not in a format the library can understand. A byte-level BPE like the RoBERTa tokenizer should have a merges files as well.

Could you try to load it in a BERT tokenizer? The BERT tokenizer saves its vocabulary as `vocab.txt` so it is possible this is the tokenizer that was used.

> It seems that the tokenizer in that directory is not a Byte level BPE, or at least not in a format the library can understand. A byte-level BPE like the RoBERTa tokenizer should have a merges files as well.
> 
> Could you try to load it in a BERT tokenizer? The BERT tokenizer saves its vocabulary as `vocab.txt` so it is possible this is the tokenizer that was used.

Thank you! It works.
Fix DataCollatorForWholeWordMask
Author: changjonathanc
Pull request: yes

# What does this PR do?

This is a quick fix for #8378 

There is another `_tensorize_batch` in other class, I think we can replace it too. 
And there might be tests missing?
@sgugger #8308

Comments:
Oh I had missed that this one was a subclass of `DataCollatorForLanguageModeling`, thanks for fixing. The last one isn't a subclass, but we can certainly change it to use the same function too!

Merging the PR to have the fix as quickly as possible, but if you want to add a test of this data collator, please go ahead! That will make the library more robust.
TPU padding in seq2seq codes 
Author: rabeehkarimimahabadi
Pull request: no

Hi
In the readme of seq2seq models https://github.com/huggingface/transformers/tree/master/examples/seq2seq it is written:

All sequences should be padded to be of equal length otherwise it leads to extremely slow training. (finetune_trainer.py does this automatically when running on TPU.)

could you point me please where in the codes this is done? and what do you mean here, isn't it like normal case that in each batch, we pad the sentence to the equal length? could you tell me the differences from what is done normally? 

thanks

Best
Rabeeh 

Comments:
I understood this is implemented in  Seq2SeqDataCollator with adding padding=longest.
Trainer QA Model Label Names
Author: ManavR123
Pull request: no

In the Trainer Class, the default label names for a QA model is ["start_positions, end_positions"]. You can see this defined on the line I linked here. Is this supposed to be the case or should it be a list with 2 strings? The tutorials for using Trainer on the transformers documentation show setting up a custom dataset with encoding having two separate keys for the start and end positions. Should we be setting the label_names parameter in our training args or is this a bug? If this is not a bug, could the documentation for setting up a custom QA dataset be updated so others don't get confused as well.
https://github.com/huggingface/transformers/blob/77a257fc210a56f1fd0d75166ecd654cf58111f3/src/transformers/trainer.py#L331

Comments:
DataCollatorForWholeWordMask error persists after fix
Author: uunal
Pull request: no

After quick fix #8379
Tried same flow of #8378
**Gives error:**
```
/usr/local/lib/python3.6/dist-packages/transformers/data/data_collator.py in __call__(self, examples)
    321         for e in examples:
    322             ref_tokens = []
--> 323             for id in e["input_ids"].tolist():
    324                 token = self.tokenizer._convert_id_to_token(id)
    325                 ref_tokens.append(token)

AttributeError: 'list' object has no attribute 'tolist'
```

When I tried with _DataCollatorForLanguageModeling_, trainer works as usual.

Comments:
The problem can be fixed  by changing `e["input_ids"].tolist()` to `e["input_ids"]`
[model card] fix md table
Author: stas00
Pull request: yes
Labels: name: model card

Just noticed, I had a borked md table that wasn't rendering as a table on the site in a few cards, this PR fixes that.

Thank you!

@julien-c
 


Comments:
Looks good now!
<img width="645" alt="Screenshot 2020-11-08 at 10 25 27" src="https://user-images.githubusercontent.com/326577/98461482-748c2380-217a-11eb-9fe4-38d6dc555233.png">
Fixed Trainer default labels for QA Model
Author: ManavR123
Pull request: yes

PR https://github.com/huggingface/transformers/pull/7191 added default labels for models. However, for QA Models it sets the default labels to ["start_positions, end_positions"] instead of ["start_positions", "end_positions"], which was the intention based on the PR description. This PR is a simple fix to this.

Fixes #8390 

@sgugger 

Comments:
Fine-tuning for QA: how to prepare custom dataset?
Author: AndreyStille
Pull request: no

I have csv dataset article,name
I want to use fine-tuning(Albert), don't undestand how to prepare my dataset to suite for fine-tuning tutorial (https://huggingface.co/transformers/custom_datasets.html#question-answering-with-squad-2-0)
Can't understand the reason why we add start and end idx of answer in context to 'answer' part of dataset.
Could I use just text in answer, without start and end ?
If context doesn't contain all answer words in a row, what indices should be selected then?


Comments:
This is for **extractive** QA which means the answer needs to be in a continuous sequence in the context. The way the model works is by predicting the start and end indices of the answer. 

> This is for **extractive** QA which means the answer needs to be in a continuous sequence in the context. The way the model works is by predicting the start and end indices of the answer.

thanks for help 
I found what I needed
Tokenizer return nothing instead of unk for certain token? 
Author: zhouhanxie
Pull request: no

## Environment info
     
- `transformers` version: 3.1.0
- Platform: pytorch
- Python version: python 3.6.9
- PyTorch version (GPU?):  1.6.0
- Tensorflow version (GPU?): na
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


## Information

Model I am using (tokenizer && model == google/electra-large-generator):

The problem arises when using:
* [x ] my own modified scripts: (give details below)
self.fill_mask  = pipeline("fill-mask", model="google/electra-large-generator",\
                                              tokenizer="google/electra-large-generator")

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x ] my own task or dataset: (give details below)
a simple fill mask type of task with transformer pipline

## To reproduce

Steps to reproduce the behavior:

1. fill_mask  = pipeline("fill-mask", model="google/electra-large-generator",\
                                              tokenizer="google/electra-large-generator")
2. fill_mask.tokenizer.tokenize(""" ̈  """)

3. output is []


## Expected behavior

In this case the missing of this punctuation would cause some downstream usage to fail due to IndexError. The problem might be intrinsic to this particular tokenizer, maybe it is worthwhile to raise a warning/error or returning a unk token? Thanks!


Comments:
Similar error at [1662]https://github.com/huggingface/transformers/issues/1662

The problem could be solved by switching to multilingual tokenizer(if present), else it would require some hot fix.
[s2s test_finetune_trainer] failing multigpu test
Author: stas00
Pull request: yes

Sam,

On a multigpu machine:
```
RUN_SLOW=1 pytest examples/seq2seq/test_finetune_trainer.py::TestFinetuneTrainer::test_finetune_trainer_slow
```
fails for me - not learning anything. 
```
>       assert first_step_stats["eval_bleu"] < last_step_stats["eval_bleu"]  # model learned nothing
E       AssertionError: assert 0.0 < 0.0
```
Looking at the logs, it gains some >0 bleu score in the first half of the epochs and then drops back to 0.00 in the last ones. On a single gpu it fluctuates between 0 and some small value - this is very fragile.

Changing to lr 3e-3 (this PR) seems to make it slightly more stable, but it could be a card specific thing - this is with rtx3090+rtx1070. So please check on your setup.

I tested that it passes with single gpu (one of each). 

Alternatively the test should compare not the first and last metrics, but perhaps something more flexible? like adding up all blue scores and checking the total >0?

But either way it feels very fragile - depending too much on the hardware type - perhaps a long term approach to make it more resilient is by feeding it more than 8 records.

@sshleifer


Comments:
Agreed that more than 8 records is the way to go. Feel free to address that in a future PR!
Update outdated TensorFlow -> PyTorch model transfer CLI example
Author: xuhdev
Pull request: yes
Labels: name: wontfix

The example is outdated and reports an error.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5296?src=pr&el=h1) Report
> Merging [#5296](https://codecov.io/gh/huggingface/transformers/pull/5296?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/7cc15bdd9675d1cec9186a8963c1f59be899ee68&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5296/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5296?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5296      +/-   ##
==========================================
- Coverage   79.29%   79.28%   -0.01%     
==========================================
  Files         138      138              
  Lines       24280    24280              
==========================================
- Hits        19252    19251       -1     
- Misses       5028     5029       +1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5296?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_transfo\_xl.py](https://codecov.io/gh/huggingface/transformers/pull/5296/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190cmFuc2ZvX3hsLnB5) | `78.74% <0.00%> (-0.20%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5296/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `79.59% <0.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5296?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5296?src=pr&el=footer). Last update [7cc15bd...65cabe0](https://codecov.io/gh/huggingface/transformers/pull/5296?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Could someone review this PR?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.