noamaanMulla-03/transformers-issues huggingface dataset:
Updated model token sizing to replace removed parameter `num_special_…
Author: jroakes
Pull request: yes
Labels: name: wontfix

…tokens`

`num_special_tokens` seems to no longer be implemented.  Replaced with `model.resize_token_embeddings(new_num_tokens=len(tokenizer))` which resizes (non-destructively, I think) the embeddings to include the new tokens.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/941?src=pr&el=h1) Report
> Merging [#941](https://codecov.io/gh/huggingface/pytorch-transformers/pull/941?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/f2a3eb987e1fc2c85320fc3849c67811f5736b50?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/941/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/941?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master     #941   +/-   ##
=======================================
  Coverage   79.04%   79.04%           
=======================================
  Files          34       34           
  Lines        6242     6242           
=======================================
  Hits         4934     4934           
  Misses       1308     1308
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/941?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/941?src=pr&el=footer). Last update [f2a3eb9...c8f622a](https://codecov.io/gh/huggingface/pytorch-transformers/pull/941?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Hi, I'm also running into this issue. Simply removing the `num_special_tokens=len(special_tokens)` argument seems to resolve the issue, since I'm able to reproduce the scores on the RocStories example.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
TFTransfoXLLMHeadModel doesn't accept lm_labels parameter
Author: tomweingarten
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): TFTransfoXLLMHeadModel

Language I am using the model on (English, Chinese....): Other

The problem arise when using:
* [ ] the official example scripts: (give details)
* [ X ] my own modified scripts: I have a script that trains a new TransformerXL

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ X ] my own task or dataset: The dataset is a language modeling dataset of novel symbolic data.

## To Reproduce

Steps to reproduce the behavior:

Call the TFTransfoXLLMHeadModel as such:
mems = transformer(data,
                                            lm_labels = lm_labels,
                                            mems = mems,
                                            training=True)

  File "/home/tom/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 891, in __call__
    outputs = self.call(cast_inputs, *args, **kwargs)
TypeError: call() got an unexpected keyword argument 'lm_labels'

If I instead include lm_labels in a dict, it is simply ignored.

## Expected behavior

The model documentation says that including lm_labels is recommended for training because it allows the adaptive softmax to be calculated more efficiently

## Environment

* OS: Ubuntu 19
* PyTorch Transformers version (or branch): 2.0.0
* Using GPU Yes


Comments:
I see now that I missed something. The documentation uses the parameter 'lm_labels' but the correct parameter is just 'labels'. The documentation says that when this parameter is present, prediction logits will not be output, but this is incorrect. They are output regardless of the presence of 'labels'.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to use BERT for ENTITY extraction from a Sequence without classification in the NER task ?
Author: ManojPrabhakar
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
My requirement here is given a sentence(sequence), I would like to just extract the entities present in the sequence without classifying them to a type in the NER task. I see that BERT has BertForTokenClassification for NER which does the classification.

So, can somebody give me an idea of how to do **entity extraction/identification using BERT**?

Comments:
I'm a bit confused: you're basically defining the broad case of named entity recognition. Is it not enough to have a binary NER (token-level classification) task for entity vs non-entity?

Assuming you have 3-class (PER, ORG, LOC) data with labels:
B-PER, I-PER, B-ORG, I-ORG, B-LOC, I-LOC, as well as O

Replace PER, ORG, LOC with ENT.
This leaves you with these labels:
B-ENT, I-ENT, O

You can do this before training, and then train a model specifically for 1-class named entity detection only.
Or you can do this as a post-processing step on the output of the normal 3-class model.

@bheinzerling Thank you!!
 I will try this.
AssertionError in official example 
Author: karajan1001
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [x] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

in the [official example](https://huggingface.co/transformers/quickstart.html) 
the tokenizer result raise an assertionError

```
text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = tokenizer.tokenize(text)

# Mask a token that we will try to predict back with `BertForMaskedLM`
masked_index = 8
tokenized_text[masked_index] = '[MASK]'
assert tokenized_text == ['[CLS]', 'who', 'was', 'jim', 'henson', '?', '[SEP]', 'jim', '[MASK]', 'was', 'a', 'puppet', '##eer', '[SEP]']

```
gives 

```
AssertionError                            Traceback (most recent call last)
<ipython-input-1-6533b2fb8252> in <module>
     16 masked_index = 8
     17 tokenized_text[masked_index] = '[MASK]'
---> 18 assert tokenized_text == ['[CLS]', 'who', 'was', 'jim', 'henson', '?', '[SEP]', 'jim', '[MASK]', 'was', 'a', 'puppet', '##eer', '[SEP]']
     19 
     20 # Convert token to vocabulary indices

AssertionError: 

```
And when i print out the tokenized_text, i find the sepcial tokens had been tokenized in a wrong way.
This may caused by the lower operation to the speicial tokens

```
print(tokenized_text)
['[', 'cl', '##s', ']', 'who', 'was', 'jim', 'henson', '[MASK]', '[', 'sep', ']', 'jim', 'henson', 'was', 'a', 'puppet', '##eer', '[', 'sep', ']']
```

## Expected behavior

no exception.

## Environment

* OS: Ubuntu 19.04 / Centos ?
* Python version: 3.7.5 / 3.6.4
* PyTorch version: 1.2.0
* PyTorch Transformers version (or branch): 2.2.1
* Using GPU ? maybe
* Distributed of parallel setup ? No
* Any other relevant information:

The official example is ok in Transformers version 2.1.1, after i update my Transformers it goes wrong

## Additional context


Comments:
Dpulicated to #2052 and closed it .
Remove pytest dependency.
Author: aaugustin
Pull request: yes

This is a follow-up to PR #2055. This file was added between the moment I wrote #2055 and the moment in was merged.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2093?src=pr&el=h1) Report
> Merging [#2093](https://codecov.io/gh/huggingface/transformers/pull/2093?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/2670b0d682746e1fe94ab9c7b4d2fd7f4af03193?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2093/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2093?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2093      +/-   ##
==========================================
- Coverage   82.67%   82.65%   -0.02%     
==========================================
  Files         111      111              
  Lines       16162    16162              
==========================================
- Hits        13362    13359       -3     
- Misses       2800     2803       +3
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2093?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/tests/optimization\_tf\_test.py](https://codecov.io/gh/huggingface/transformers/pull/2093/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL29wdGltaXphdGlvbl90Zl90ZXN0LnB5) | `86.76% <100%> (ø)` | :arrow_up: |
| [transformers/tests/modeling\_tf\_common\_test.py](https://codecov.io/gh/huggingface/transformers/pull/2093/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX3RmX2NvbW1vbl90ZXN0LnB5) | `95.38% <0%> (-1.54%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2093?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2093?src=pr&el=footer). Last update [2670b0d...010489c](https://codecov.io/gh/huggingface/transformers/pull/2093?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@julien-c Could you merge this please? :-)
Confusing tokenizer result on single word
Author: malmaud
Pull request: no
Labels: name: wontfix

Not sure if this is expected, but it seems confusing to me: 

```python
import transformers
t=transformers.AutoTokenizer.from_pretrained('roberta-base')
t.tokenize("mystery")
```

yields two tokens, `['my', 'stery']`.

Yet

```
t.tokenize("a mystery") 
```

*also* yields two tokens, `['a', 'Ġmystery']`. I would have thought this should yield one more token than tokenizing "mystery" alone.


Comments:
Hey @malmaud I think this #1196 can help you. The Roberta/GPT2 tokenizer expect a space to start. Without that, it sounds like you'll get strange behaviors.

To get the same output, in your first example, change it to 
```
t.tokenize("mystery", add_prefix_space=True)
['Ġmystery']
```

That does work, thanks. I'm still confused why this doesn't work, though:

```
t.tokenize("
 mystery 
")
```

gives `['
', 'my', 'stery', '
']`

Hey @malmaud, spent some time going through the source code. So like above this gives the correct result:
```
t.tokenize("mystery", add_prefix_space=True)
['Ġmystery']
```
However
```
t.tokenizer(" mystery")
['my', 'stery']
```
I thought these should be doing the same thing. In the tokenization_gpt2.py file, it says:
```
if add_prefix_space:
    text = ' ' + text
```
This should give the same results in both files then however when I add a print(text) statement before and after that I noticed I got these results. (using your example now)
```
t.tokenize("
 mystery")
mystery
mystery
['
', 'my', 'stery']

t.tokenize("
 mystery", add_prefix_space=True)
mystery
 mystery
['
', 'Ġmystery']
```

This means that even though we are putting a single word in with a leading space, something in the preprocessing is getting rid of the initial space(s). So we need to use the add_prefix_space=True in order to get the space back or else the function won't be using the string we are expecting it will be using.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
❓ How to use cached hidden states in run_generation ?
Author: astariul
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

https://github.com/huggingface/transformers/blob/5c3b32d44d0164aaa9b91405f48e53cf53a82b35/examples/run_generation.py#L124

This line states that we could use `cached hidden states`. Correct me if I'm wrong :

* **Without using `cached hidden states`** : every step, the next token is predicted, but also all previous tokens are re-computed (which is useless because we already predicted it !)
* **Using `cached hidden states`** : every step, the next token is predicted, but previous tokens are not re-computed, because we are using their cached states.

So using cached hidden states would greatly increase the inference speed, specially for long generations.

---

My question is : **How to do that ?**

From the documentation I understand how to get the `cached hidden states` from the forward pass of the model, but I don't understand how to use it at the following step ?

Comments:
Hi! Yes, you understood the gist of it. The self-attention related to already computed tokens is not computed again.

In order to use the past, you would get the past from the model pass (I'm using GPT-2 in this example, XLNet would have `mems` instead of `past`):

```py
logits, past = model(**inputs)
```

and you would then use  the past on the following pass as follow:

```py
logits, past = model(**inputs, past=past)
```

Thank you for your fast response @LysandreJik !

Now it's very clear, but I have one more question :

For XLNet and TransfoXL, we need to use memory in order to not recompute previously generated token. This is ok when not using the memory for something else.

**But what if the memory is already used for something else ?**
Like we have a memory of 256 for XLNet, representing previous segments (or whatever), if we update the memory everytime a new token is generated, it means we are loosing part of the memory (and after generating 256 tokens, we will not be able to see anymore the memory of previous segment !).

**Is there a way around this problem in the current API ?**

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Will BERT weights for SQuAD be released?
Author: lcswillems
Pull request: no

Hi,

Are you going to release the weights after training on SQuAD 2.0?
Thank you for your great work.

Best,
Lucas Willems

Comments:
Hi Lucas, probably not.

The goal of this repository is to provide easy access to pretrained model for transfer learning research. 

Providing downstream task models will make us handle a combinatory explosion of combinations to provide the various pretrained BERT models fine-tuned on each GLUE/SQuAD task with hyper-parameters optimization and all the relevant adaptation decision that are still mostly open research questions.

But we do provide examples for fine-tuning that gives decent results and can be trained in a reasonable time on standard cloud compute.

@lcswillems were you able to find the weights anywhere else? 

Looks like HF released them after all

https://huggingface.co/transformers/pretrained_models.html
With GPT-2 is it possible to get previous word prediction?
Author: hypnoai
Pull request: no


Feature/Question: With GPT-2 is it possible to get previous word prediction?

Hi,

I say this after seeing this https://towardsdatascience.com/deconstructing-bert-distilling-6-patterns-from-100-million-parameters-b49113672f77

And wondering how I could maybe write a method that would allow me to predict the previous word? (ideally for GPT2)

Many thanks,
Vince.

Comments:
Hi! There is one big difference between BERT and GPT-2, in that BERT is trained using masked language modeling, whereas GPT-2 is trained using causal language modeling.

During pre-training, BERT learns to predict masked words given a bi-directional context. GPT-2, on the other hand, learns to predict a word given only its left context. This is why GPT-2 is very good at text generation (it only needs the left-hand side context), while BERT isn't.

Given this,  GPT-2 won't be able to do previous word prediction, as it does not handle the right-hand side context.

If you want to train your own GPT-2 model to predict previous words, you could feed in your entire training set in reverse word order.  Then GPT-2 would learn to predict text backwards, and that model would then be able to tell you what word should come before a piece of text.
How to use BERT or word embedding for e-commerce product classification. 
Author: Raghavendra15
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I want to classify products on an e-commerce site. For example, if we take Amazon for example if a product name is iPhone XS then it should be categorized to Electronics -> mobile, very straight forward, however, the problem comes when we train the model on clothes and many other sports items. 
For example: "George - George Men's Cargo Short" found on walmart is being classified as SPORTS & OUTDOOR, FASHION. However, it should be classified to Clothes.

Currently, we have tried text CNN but I'm very positive that BERT or other word embeddings can enhance the performance. 

Base Code: 
https://github.com/brightmart/text_classification
However, it appears that TextCNN is better than BERT as per the author of this repository. Does anyone know what's the ideal way to approach this problem?
![image](https://user-images.githubusercontent.com/7957331/64069995-c22b0900-cc24-11e9-8077-21313e38c970.png)





Comments:
you solve this problem?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Extending `examples/` to TensorFlow
Author: HanGuo97
Pull request: no
Labels: name: wontfix

## 🚀 Feature

Hi, thanks for putting in the tremendous effort for TensorFlow-PyTorch interoperability! Would those scripts in the `examples/` be soon extended to Tensorflow as well?

## Motivation

I (and presumably many others) rely on the examples to quickly experiment with models and ideas. Extending the examples to Tensorflow would be hugely helpful, and should help the codebase reach a broader set of audiences.

## Additional context

<!-- Add any other context or screenshots about the feature request here. -->

Comments:
Indeed, there is currently one example for tensorflow, `run_tf_glue` and it doesn't have command-line arguments. We'll update this one to make it as flexible as the PyTorch one and add other examples when we have the bandwidth.

Do you want to help in this project? Happy to welcome a PR on this topic (for instance to add command line argument similar to `run_glue` in `run_tf_glue`).

Thanks for the response. I'm not an expert in this field but I'm happy to help and review codes.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Defining Models in TF 2.0 and Extending Them
Author: vyraun
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Hi,

Thanks for the awesome library 😊

I saw the examples on fine-tuning the models. My question is, how could we get model definitions i.e the layered architectures (model.summary) in Keras.

Any example notebooks demonstrating how we could get the model definitions and extend the architectures (by subclassing or manually tweaking the layers)?

Cheers,
Vikas

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to preprocess my own data to use RoBERTa of Multiple GPUs
Author: gr8Adakron
Pull request: no
Labels: name: wontfix

Hey,

I am bit naive using deep learning of text-classification, my data **(.csv)** consist of basically two columns:

- Text
- Labels

As per basic objective, model should take unseen text and predict label _(variable y)_ from the trained labels.

**I followed this tutorial to train RoBERTa algorithm:**
- [https://colab.research.google.com/drive/1xg4UMQmXjDik3v9w-dAsk4kq7dXX_0Fm](https://colab.research.google.com/drive/1xg4UMQmXjDik3v9w-dAsk4kq7dXX_0Fm)

Here the input format is universal (Train.tsv and Test.tsv) with 2 columns (which I mentioned above). The only problem is that this code doesn't utilize the multiple GPUs **(I even tried the DataParallel wrapper)**. 
Somehow, I found the repository of the pytorch-transformer where they have given an example of how to utilize multiple GPUs and train the RoBERTa model i.e:

- [https://github.com/huggingface/pytorch-transformers/tree/master/examples](https://github.com/huggingface/pytorch-transformers/tree/master/examples)
This repository example takes the input-data in the wiki.text data format and they have provided the link [ Pretraining RoBERTa using your own data ](https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.pretraining.md#pretraining-roberta-using-your-own-data) and **yet I don't find it of any use as it just talks about the format what they think is right not about the transforming of the standard format (i.e: .CSV)**

Here is the sample:
```
 = Valkyria Chronicles III = 
 
 Senjō no Valkyria 3 : <unk> Chronicles ( Japanese : 戦場のヴァルキュリア3 , lit . Valkyria of the Battlefield 3 ) , commonly referred to as Valkyria Chronicles III outside Japan , is a tactical role @-@ playing video game developed by Sega and Media.Vision for the PlayStation Portable . Released in January 2011 in Japan , it is the third game in the Valkyria series . <unk> the same fusion of tactical and real @-@ time gameplay as its predecessors , the story runs parallel to the first game and follows the " Nameless " , a penal military unit serving the nation of Gallia during the Second Europan War who perform secret black operations and are pitted against the Imperial unit " <unk> Raven " . 

```

I mean, why? do they have to use this format, why can't they go with standard format of classification algorithm? and if they want to then why they don't provide the detail regarding the data and adjusting the personal data corresponding to the algorithm intake?

Just wanted to know, how to preprocess the data according to multiple GPUs RoBERTa example, any help would be appreciated.

thanks.


Comments:
@spolu @cynthia @myleott 

Hi, you can follow the `run_glue` example which is better for text classification.
But you will have to modify it for your needs, it's not plug and play.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.