noamaanMulla-03/transformers-issues huggingface dataset:
which special token is used to predict the score in roberta?
Author: tzhxs
Pull request: no

In bert, we use the embedding of <cls> to predict the score, how about the roberta?

Comments:
Can you give some more information? It's not clear what you mean by "score". The special classification token for RoBERTa is `
`.

Thanks!

@tzhxs If that's everything you need, please close this topic.

ok
doc: fix pretrained models table
Author: stefan-it
Pull request: yes

Hi,

this PR fixes the pretrained models table, see #2145.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2146?src=pr&el=h1) Report
> Merging [#2146](https://codecov.io/gh/huggingface/transformers/pull/2146?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/2e2f9fed554bb5f147ea3d9573004b447dd7c9e7?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2146/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2146?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2146   +/-   ##
=======================================
  Coverage   80.07%   80.07%           
=======================================
  Files         112      112           
  Lines       16866    16866           
=======================================
  Hits        13505    13505           
  Misses       3361     3361
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2146?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2146?src=pr&el=footer). Last update [2e2f9fe...c852efa](https://codecov.io/gh/huggingface/transformers/pull/2146?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great, thanks @stefan-it
the docs pretrained models is missing
Author: PiotrCzapla
Pull request: no

Hi, 
your docs have the table with pretrained models missing, probably some formatting error, as the source code has the table
<img width="1280" alt="Screenshot 2019-12-11 at 17 31 01" src="https://user-images.githubusercontent.com/340180/70640463-4174a380-1c3c-11ea-9c6e-ca343ef46332.png">
https://huggingface.co/transformers/pretrained_models.html
Cheers,
Piotr


Comments:
Should be working now :)

Thanks @PiotrCzapla for raising the issue, @stefan-it fixed it earlier today!
How to detokenize a BertTokenizer output?
Author: bprabhakar
Pull request: no

I was wondering if there's a proper way of detokenizing the output tokens, i.e., constructing the sentence back from the tokens? Considering the fact that the word-piece tokenisation introduces lots of `#`s.

Comments:
You can remove ' ##' but you cannot know if there was a space around punctuations tokens or uppercase words.

Yes. I don't plan to include a reverse conversion of tokens in the tokenizer.
For an example on how to keep track of the original characters position, please read the `run_squad.py` example.

In my case, I do: 
```
tokens = ['[UNK]', '[CLS]', '[SEP]', 'want', '##ed', 'wa', 'un', 'runn', '##ing', ',']
text = ' '.join([x for x in tokens])
fine_text = text.replace(' ##', '')
```


Apostrophe is considered as a punctuation mark, but often it is an integrated part of the word.  Regular `.tokenize()` always converts apostrophe to the stand alone token, so the information to which word it belongs is lost. If the original sentence contains apostrophes, it is impossible to recreate the original sentence from its' tokens (for example when apostrophe is a last symbol in some word `convert_tokens_to_string()` will join it with the following one). In order to overcome this, one can check the surroundings of the apostrophe and add `##` immediately after the tokenization.  For example:
```
sent = "The Smiths' used their son's car" 
tokens = tokenizer.tokenize(sent)
```
now if you fix `tokens` to look like:

**original** `=>['the', 'smith', '##s', "'",     'used', 'their', 'son', "'",     's',      'car']`
**fixed** `  => ['the', 'smith', '##s', "##'", 'used', 'their', 'son', "##'", '##s', 'car']`

you will be able to restore the original words.


@thomwolf could you point to the specific section of `run_squad.py` that handles this, I'm having trouble

EDIT: is it this bit from `processors/squad.py`? 
```python
tok_to_orig_index = []
        orig_to_tok_index = []
        all_doc_tokens = []
        for (i, token) in enumerate(example.doc_tokens):
            orig_to_tok_index.append(len(all_doc_tokens))
            sub_tokens = tokenizer.tokenize(token)
            for sub_token in sub_tokens:
                tok_to_orig_index.append(i)
                all_doc_tokens.append(sub_token)
```
RoBERTa tokenization: Why do we call 'all_special_tokens' in each tokenize loop?
Author: volker42maru
Pull request: no

Is there a reason why the property function`all_special_tokens` is called in each iteration in `tokenize()` when looping over all tokens?

When I initialize a new variable and call `all_special_tokens` only once in the tokenizer init, the tokenization is speed-up around 2~3 times for me.

Comments:
AttributeError: module 'tensorflow.python.training.training' has no attribute 'list_variables'
Author: Dhanachandra
Pull request: no
Labels: name: wontfix

TF version 1.1.0:

convert_tf_checkpoint_to_pytorch("../biobert1.1/biobert_v1.1_pubmed/biobert_model.ckpt",
                                     "../biobert1.1/biobert_v1.1_pubmed/bert_config.json",
                                     "../biobert1.1/pytorch_model")

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


You just have to upgrade your tensorflow
Extracting Features Example
Author: SaschaStenger
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Hello. 

Sorry, if my question is out of date or i just didn't find it, but i'm looking for the example/extract_features.py 
that was supposed to be in in this repo (as mentioned in this stackoverflow  [post](https://stackoverflow.com/questions/55369821/how-to-train-a-neural-network-model-with-bert-embeddings-instead-of-static-embed)) and can't find it anymore. Was it just in an earlier release and got scrapped?

Thank you in advance for any help

Comments:
Yes it was removed from the repo. I think we’ll add it again (and update it
to pytorch-transformers) since several people have been missing it.

Cc @LysandreJik

On Tue, 27 Aug 2019 at 17:21, Sascha Stenger <notifications@github.com>
wrote:

> ❓ Questions & Help
>
> Hello.
>
> Sorry, if my question is out of date or i just didn't find it, but i'm
> looking for the example/extract_features.py
> that was supposed to be in in this repo (as mentioned in this
> stackoverflow post
> <https://stackoverflow.com/questions/55369821/how-to-train-a-neural-network-model-with-bert-embeddings-instead-of-static-embed>)
> and can't find it anymore. Was it just in an earlier release and got
> scrapped?
>
> Thank you in advance for any help
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/pytorch-transformers/issues/1123?email_source=notifications&email_token=ABYDIHI6ESI3KSNXXUQZFZDQGVA7BA5CNFSM4IQFZBAKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HHV4OEQ>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABYDIHMWWEDMXVWDLZIX4NDQGVA7BANCNFSM4IQFZBAA>
> .
>


That's nice to hear. Thank you very much 

Any update on the extract_features script?


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
GPT-2 language model decoding method
Author: haozheji
Pull request: no

I am wondering what is the official decoding method when evaluating the language model? The  doc says `run_gpt2.py` implement the beam-search. While to me, it seems it's still greedy search with sampling. 

Comments:
`run_gpt2` has top-K which is better than beam-search for high-entropy tasks like open-domain generation. The coming release example (currently on the xlnet branch to be merged with master on Monday) will have top-K and Nucleus sampling (see Holtzman et al. http://arxiv.org/abs/1904.09751)

Hi,
Is it possible to include beam search decoding in ```run_generation.py``` ?

hope that beam search appears in run_generation.py

We'll add it.
cc @rlouf 

@thomwolf I see that run_generation.py has disappeared and beam_search does not exist anymore, nor in transformers/generate. Where could we find the implementation of batch beam_search in this repo ?

You can’t... so far. We are reworking the API for greedy decoding and sampling, and will work on beam search afterwards.
Fix typo in examples/run_glue.py args declaration.
Author: adelevie
Pull request: yes

deay -> decay

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2143?src=pr&el=h1) Report
> Merging [#2143](https://codecov.io/gh/huggingface/transformers/pull/2143?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/4c12860f7ae61659aed2675498350a386fc4e122?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2143/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2143?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2143   +/-   ##
=======================================
  Coverage   80.07%   80.07%           
=======================================
  Files         112      112           
  Lines       16867    16867           
=======================================
  Hits        13506    13506           
  Misses       3361     3361
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2143?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2143?src=pr&el=footer). Last update [4c12860...059111d](https://codecov.io/gh/huggingface/transformers/pull/2143?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great, thanks!
Change gpt2 language model loss function
Author: alecalma
Pull request: no
Labels: name: wontfix

Hi all,

I want to include a new loss term for the gpt2 training loss. I am using the script run_lm_finetuning from the examples. This is my command:

python examples/run_lm_finetuning.py --output_dir=output --model_type=gpt2 --model_name_or_path=gpt2 --do_train --train_data_file=$TRAIN_FILE --eval_data_file=$TEST_FILE --overwrite_output_dir --max_steps 50


 but I really can't figure out which loss function is being used. If i print inside the GPT2LMHeadModel forward function, nothing happens. Could you please tell me which loss function should I change?

Thank you a lot.

Comments:
Afaik the "default" loss function that gets computed if you pass your labels to `GPT2LMHeadModel` is `torch.nn.CrossEntropyLoss`. If you want to use a different loss function, can't you just grab the logits from the model and apply your own?

Source:
https://github.com/huggingface/transformers/blob/391db836ab7ed2ca61c51a7cf1b135b6ab92be58/transformers/modeling_gpt2.py#L539

Unfortunately if I print a string inside the forward function, and then I run the training script, I don't get anything printed, so it seems like the training script is not using that function at all.

Hello! The `GPT2LMHeadModel` does have a way to compute its own cross-entropy loss, but only when the `labels` are specified -> you're providing the values like so:
```
model(inputs, labels=inputs)
```
and the model takes care of shifting the inputs to calculate a causal language modeling loss on them with cross-entropy.

If you wish to use your own loss function, don't specify the labels and the model will return a tuple containing the language modeling logits as the first value.

Hi,

thanks for your answer.
Can you tell me why a print statement inside the forward fuction of the GPT2LMHeadModel doesn't print anything when I run the run_lm_finetuning script ?

Which is the forward function I need to change?

Thanks.

Where have you put your print statement? Do you have `transformers` installed in your environment or is it relying on the cloned repository? You could try to add a breakpoint and debug it to see which function calls are made and how the loss is calculated.

Once again, if you wish to use your own loss function, don't specify the labels and the model will return a tuple containing the language modeling logits as the first value.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
about the special tokens
Author: tzhxs
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
The question about roberta.
I konw that Bert use the embedding of token 'cls' to do predict, but when it comes to roberta, I dont know it clearly. Can you tell me which token's embedding is used to do predict in this project? Is it '
' ?

Comments:
Please close this. It's a duplicated of your other question.
Where is extract_features.py and run_classifier.py ?
Author: JiangYanting
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hello! I couldn't find the extract_features.py and run_classifier.py. Have they been renamed ?

Comments:
Reading the answer given by @thomwolf in #1123, I'm sure that `extract_features.py` script has been removed from repo, but in the future it could be updated!

Reading the answer given by @ningjize in #1011, I'm sure that `run_classifier.py` script has been updated as `run_glue.py` script, that you can find in `examples/` directory [here](https://github.com/huggingface/transformers/blob/master/examples/run_glue.py).

> ## ❓ Questions & Help
> Hello! I couldn't find the extract_features.py and run_classifier.py. Have they been renamed ?
How to replicate Arxiv-NLP but for different subject?
Author: arxivcrawler
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Hi, I'm fairly new to NLP so apologies for my ignorance on some things.

If I wanted to fine tune text generation on a subject matter ( like Harry Potter), how would I do that?

Im looking to use XLNET and it seems like there isn't any support for fine tuning for that model.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[cli] Upload is now compatible with folders
Author: julien-c
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2172?src=pr&el=h1) Report
> Merging [#2172](https://codecov.io/gh/huggingface/transformers/pull/2172?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/d46147294852694d1dc701c72b9053ff2e726265?src=pr&el=desc) will **increase** coverage by `0.48%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2172/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2172?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2172      +/-   ##
==========================================
+ Coverage   80.32%   80.81%   +0.48%     
==========================================
  Files         114      113       -1     
  Lines       17102    16999     -103     
==========================================
  Hits        13737    13737              
+ Misses       3365     3262     -103
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2172?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/2172/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2RhdGEvcHJvY2Vzc29ycy9zcXVhZC5weQ==) | `14.49% <0%> (+0.3%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2172?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2172?src=pr&el=footer). Last update [d461472...fb92209](https://codecov.io/gh/huggingface/transformers/pull/2172?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
MemoryError on run_lm_finetuning.py
Author: echan00
Pull request: no
Labels: name: wontfix

Previous versions of finetune_on_pregenerated.py had a `--reduce_memory` parameter to keep memory requirements from going overboard, it seems like it is no longer available in the new run_lm_finetuning.py file?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I also have the same problem....
AttributeError: 'tuple' object has no attribute 'softmax'
Author: Raghavendra15
Pull request: no

I get the following error when I use the pytorch transformers, It used to work just fine in the previous pretrained-bert,
Original code: https://github.com/ceshine/pytorch-pretrained-BERT/blob/master/notebooks/Next%20Sentence%20Prediction.ipynb

Code which has error:
model.eval()

res = []

mb = progress_bar(eval_dataloader)
for input_ids, input_mask, segment_ids in mb:
    input_ids = input_ids.to(device)
    input_mask = input_mask.to(device)
    segment_ids = segment_ids.to(device)

    with torch.no_grad():
        res.append(nn.functional.softmax(
            model(input_ids, segment_ids, input_mask), dim=1
        )[:, 0].detach().cpu().numpy())
        
res = np.concatenate(res)


Error stacktrace:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-37-a816e24060d8> in <module>
     24     with torch.no_grad():
     25         res.append(nn.functional.softmax(
---> 26             model(input_ids, segment_ids, input_mask), dim=1
     27         )[:, 0].detach().cpu().numpy())
     28 

/common/users/rs1693/my_venv/venv_bert/lib64/python3.6/site-packages/torch/nn/functional.py in softmax(input, dim, _stacklevel, dtype)
   1261         dim = _get_softmax_dim('softmax', input.dim(), _stacklevel)
   1262     if dtype is None:
-> 1263         ret = input.softmax(dim)
   1264     else:
   1265         ret = input.softmax(dim, dtype=dtype)

AttributeError: 'tuple' object has no attribute 'softmax'

I read many posts where they say to do the following:(But not sure where in the code I have to make these changes)
1. disable aux_logits when the model is created here by also passing aux_logits=False to the inception_v3 function.

2. Edit your train function to accept and unpack the returned tuple here to be something like:

output, aux = model(input_var)

But where in the above function I have to do this?


Comments:
HI,
I have the same problem!
What was the solution here?

Me too.

Need more information like version of python/pytorch/transformers (all the information requested in the issue templates actually)

> Need more information like version of python/pytorch/transformers (all the information requested in the issue templates actually)

I am experiencing this issue as well with the BertForNextSentencePrediction model and not having much luck with a solution. I'm using macOS Mojave 10.14.6, python 3.7, pytorch 1.3.1 and transformers 2.2.1. 

Please let me know if there is any more details I can provide.  Thanks!

You should open a new issue with a clean code example we can test and the associate full error message.
xlm-mlm-100-1280 model is not available for download
Author: jensdebruijn
Pull request: no
Labels: name: wontfix

xlm-mlm-100-1280 model is not available for download, see:
https://s3.amazonaws.com/models.huggingface.co/bert/xlm-mlm-100-1280-tf_model.h5

The model for pytorch is available



Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Visualizing the Inner Workings of Attention 
Author: alshahrani2030
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

what should I DO to plot my  model by BertViz tool??
 i am useing config = BertConfig.from_pretrained(“bert-base-uncased”,output_attentions=True,output_hidden_states=True, num_labels=2)
model = BertForSequenceClassification.from_pretrained(“bert-base-uncased”, config= config)

Thank you in advance
<!-- A clear and concise description of the question. -->


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Model does not train when using new BertModel, but does with old BertModel
Author: Peter-Devine
Pull request: no
Labels: name: wontfix

## 📚 Migration

I am currently working on using Transformers with Snorkel's classification library [https://github.com/snorkel-team/snorkel](https://github.com/snorkel-team/snorkel) (for MTL learning in the future). I currently am trying to troubleshoot why the model is not learning, and so have my experiment set up such that the Snorkel library learns one task, essentially training a BERT model and linear layer.

The code for this experiment can be found at [https://github.com/Peter-Devine/test_cls_snorkel_mtl]( https://github.com/Peter-Devine/test_cls_snorkel_mtl ). To run it, you will need torch, snorkel, numpy, pytorch_pretrained_bert and transformers.

My problem is as follows.
When I run the code in `test_cls_snorkel_mtl/tutorials/ISEAR_pretrain_tutorial.py`, my code runs fine and the model's validation accuracy scores are good. This is because I am using the old pytorch_pretrained_bert BertModel in `test_cls_snorkel_mtl/modules/bert_module.py`. If you uncomment line 6 of `test_cls_snorkel_mtl/modules/bert_module.py` and use the new transformers BertModel, then running `test_cls_snorkel_mtl/tutorials/ISEAR_pretrain_tutorial.py` will result in a model that never converges and bad validation accuracy.

From reading the code on Snorkel, I cannot seem to find the reason as to why this would be. What are the major changes in training a model between versions of pytorch_pretrained_bert and transformers. Do back-passes etc. work the same way in both models?

Thanks

Comments:
You can check the two migration guides, they explain all the differences:
- https://github.com/huggingface/transformers#Migrating-from-pytorch-transformers-to-transformers
- https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.