noamaanMulla-03/transformers-issues huggingface dataset:
Change tensorboard imports to use built-in tensorboard if available
Author: bilal2vec
Pull request: yes

Related issue: #1427 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1454?src=pr&el=h1) Report
> Merging [#1454](https://codecov.io/gh/huggingface/transformers/pull/1454?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/d688af19e5ce92c1395820a89e3f3b635eacc2ba?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1454/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1454?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1454   +/-   ##
=======================================
  Coverage   84.72%   84.72%           
=======================================
  Files          84       84           
  Lines       12591    12591           
=======================================
  Hits        10668    10668           
  Misses       1923     1923
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1454?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1454?src=pr&el=footer). Last update [d688af1...5ce8d29](https://codecov.io/gh/huggingface/transformers/pull/1454?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Fine with me, thanks!

My critique would be that just writing a general 'except' is not PEP-y. The correct error test should be checked. Then again, large parts of the whole package are not PEP-y so it might not be important for the developers.
Issues in visualizing a fine tuned model
Author: chikubee
Pull request: no
Labels: name: wontfix

BertModel finetuned for a sequence classification task does not give expected results on visualisation.
Ideally, the pretrained model should be loaded into BertForSequenceClassification, but that model does not return attentions scores for visualisation.
When loaded into BertModel (0 to 11 layers), I assume the 11th layer (right before classification layer in BertForSequenceClassification) is the right layer to check attention distribution.
But every word is attentive to every other word.
I am wondering what can be the possible reasons and how I can fix it.
Thanks.

<img width="753" alt="Screenshot 2019-07-30 at 11 19 46 AM" src="https://user-images.githubusercontent.com/25073753/62104050-08a2e600-b2bc-11e9-889a-88a6c0c9e2ea.png">


Comments:
"But every word is attentive to every other word." --> I don't think that's an error, that's the general way how attention mechanism works. But definitely weights of these attentions to a particular word would vary and based on these weighted attentions and other contextual info. the downstream tasks (entailment, prediction, classification etc.) would be performed. I haven't worked on attention viz yet, but I think checking [BertViz](https://github.com/jessevig/bertviz) repo. or posting your issue over there would be more fruitful. 

I don't know which framework you use for visualizing attention so I can't really help but a way to make the model output attention weights is by loading it like that:
```
model = BertModel.from_pretrained("bert-base-uncased", state_dict=model_state_dict, output_attentions=True).
```

The model will then output a tuple with the last element being the full list of attentions weights (see docstring and doc of the model).

@thomwolf I see similar results with output_attentions=True. 
The output predictions are correct but the attention scores are comparatively higher than in the case with pretrained model ('bert-base-uncased'). This makes me think, if the attention scores extracted are even correct or not for a fine tuned model.

I used [BertViz](https://github.com/jessevig/bertviz/tree/master/bertviz) as a tool for visualization.

Hi @chikubee , how does the visualization of lower layers look like? First of all, I don't think visualizing final layers is a good idea. I also tried BertVis, and I found that attention weights of higher layers are usually more uniformly distributed, like the screenshot you provided. Although I used it for Transformer encoder-decoder, but I think the phenomenon is similar. Would be interested to see your lower layer visualization.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Write with Transformer: Changing settings on Mobile?
Author: varkarrus
Pull request: no

## ❓ Questions & Help

It's great to see new features and options, in particular the Max Time option to generate longer outputs. However, none of the Model Settings are available on mobile...?

In order to change the model settings on mobile, I had to download Firefox, mess with the CSS settings in about://config to make everything really tiny, zoom in on the extremely small settings box, slide things around, then set everything back, and I'd have to do most of those all over again if I end up closing / reloading the tab.


Comments:
You're right, the interface isn't well suited to mess with settings on mobile. It's on our roadmap!

awesome! I found a dumb workaround; saved a copy of the page, changed the default values, then put it in my dropbox!

Haha that’s a great hack!

Closing this for now, thanks
Fine tuning GPT-2 for LM objective function
Author: bakszero
Pull request: no
Labels: name: wontfix

I'm looking for finetuning GPT-2 parameters for a custom piece of text, so that the weights are tuned for this piece of text, building from the initial model.

The script here does it for the original tensorflow implementation:  https://github.com/nshepperd/gpt-2 , could you please give me suggestions on how to do this finetuning from the Pytorch version and subsequently use it for text generation? It'd be of great help, thanks!

Comments:
Seems like this is now possible with last week's [merged PR](https://github.com/huggingface/pytorch-pretrained-BERT/pull/597), but I'm curious to see what the core devs say about this as well (btw, keep up the great work!)

I have the same question :)
I have tried the codes for BERT finetuning which is in lm-finetuning folder but looking for the same script for gpt-2.

Thanks

Yes fine-tuning GPT-2 is fixed with #597 indeed.
I'll see if I can add an example but basically changing `gpt` to `gpt-2` in the gpt example should be pretty much fine.

@thomwolf Thanks for the great work.
just wondering in order to do unsupervised LM fine-tuning (not classification) on a new dataset, should we just modify run_openai_gpt.py or is there an existing script for that?

No existing script for that but you can start from run_openai indeed and use just the `OpenAIGPTLMHeadModel`.
If you want to supply another example, happy to welcome a PR

I am still confused as to how to use the run_openai_gpt.py to finetune gpt2 model. A short example would be helpful

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Torchscript Trace slower with C++ runtime environment.
Author: sukuya
Pull request: no
Labels: name: wontfix

I traced the BERT model from PyTorchTransformers library and getting the following results for 10 iterations.
a) Using Python runtime for running the forward: 979,292 µs

```
import time
model = torch.jit.load('models_backup/2_2.pt')
x = torch.randint(2000, (1, 14), dtype=torch.long, device='cpu')
start = time.time()
for i in range(10):
    model(x)
end = time.time()
print((end - start)*1000000, "µs")
```
b) Using C++ runtime for running the forward: 3,333,758 µs which is almost 3x of what Python

  ```
torch::Tensor x = torch::randint(index_max, {1, inputsize}, torch::dtype(torch::kInt64).device(torch::kCPU));
  input.push_back(x);
  #endif
  // Execute the model and turn its output into a tensor.
  auto outputs = module->forward(input).toTuple();
  auto start = chrono::steady_clock::now();
  for (int16_t i = 0; i<10; ++i)
  {
    outputs = module->forward(input).toTuple();
  }
  auto end = chrono::steady_clock::now();
  cout << "Elapsed time in microseconds : " 
		<< chrono::duration_cast<chrono::microseconds>(end - start).count()
		<< " µs" << endl;
```
@thomwolf  any suggestions on what am I missing ? 

Comments:
2 possible reasons:
1. the first time you run `forward` will do some preheating work, maybe you should exclude the first run.
2. try exclude `toTuple`

According to my experience, jit with python or c++ will cost almost the same time.

@Meteorix Forward is called once before the loop, are you talking about something else. 
Excluding `toTuple` doesn't help. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Unclear error message when unable to cache the model
Author: czyzby
Pull request: no

I encountered the following error:
```
[2019-05-07 11:06:51,904: ERROR/ForkPoolWorker-1] Model name 'bert-base-uncased'
was not found in model name list (bert-base-uncased, bert-large-uncased, bert-base-cased,
bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased,
bert-base-chinese).
We assumed 'https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased.tar.gz'
was a path or url but couldn't find any file associated to this path or url. 
```

After some debugging, I found that the root cause of the issue was the fact that the application is unable to cache the model in the home directory. It was a simple I/O error rather than an issue with the model name or file downloading, as the message suggests. I think it would be worth it to handle this case with an appropriate message, and what's more important - throwing an exception.

In my case, I did get the error logs, but the application initiated "successfully" - with the tokenizer and model set to `None`. If the library is not able to load the model for any reason, I'd expect it to throw an exception rather than just (almost) silently return a `None`.

Comments:
Yes, this error message hides several potential sources, I'll see if I can disentangle the error messages :) 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This is still an issue. I suggest to improve the message and raise an exception if unable to load any of the models, instead of silently returning `None`.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Still an issue, as far as I know.

@czyzby Hello，How do you solve the cache problem? I have the same problem but can't fix it

@twothousand To be honest, I don't remember, but I think the directory did not exist or lacked write permission. Are you sure the cache is causing the problem? If you changed the cache directory, make sure the folder exists and has appropriate permissions - otherwise I'd debug model loading and see which exception is being ignored.

@thomwolf It seems that it's still an issue, will you look into proper error handling?

Yes, it should have been improved on the latest release 2.1.1 (with the merge of #1480).

Maybe open a new issue with clear details of the current issue?
Any idea how to use pytorch-transformers for Entity Linking?
Author: almoslmi
Pull request: no
Labels: name: wontfix

Thanks @huggingface for such a great library.
I am interested to use pytorch-transformers for entity linking. Any idea how to that?
Any help in this regard is highly appreciated.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
the working example code to use BertForQuestionAnswering 
Author: jeffxtang
Pull request: yes

so we can use the pre-trained and fine-tuned on SQUAD Bert model to get an answer from a text and a question, similar to the way the CoreML model BERTSQUADFP16.mlmodel is used in the iOS example [Finding Answers to Questions in a Text Document](https://developer.apple.com/documentation/coreml/finding_answers_to_questions_in_a_text_document)

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1502?src=pr&el=h1) Report
> Merging [#1502](https://codecov.io/gh/huggingface/transformers/pull/1502?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a701c9b32126f1e6974d9fcb3a5c3700527d8559?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1502/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1502?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1502      +/-   ##
==========================================
+ Coverage   85.98%   85.98%   +<.01%     
==========================================
  Files          91       91              
  Lines       13579    13574       -5     
==========================================
- Hits        11676    11672       -4     
+ Misses       1903     1902       -1
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1502?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/1502/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2JlcnQucHk=) | `88.17% <ø> (ø)` | :arrow_up: |
| [transformers/tests/modeling\_tf\_common\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1502/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX3RmX2NvbW1vbl90ZXN0LnB5) | `94.79% <0%> (-1.74%)` | :arrow_down: |
| [transformers/modeling\_tf\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/1502/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3RmX2F1dG8ucHk=) | `53.33% <0%> (+2.08%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1502?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1502?src=pr&el=footer). Last update [a701c9b...e76d715](https://codecov.io/gh/huggingface/transformers/pull/1502?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Nice, thanks!
bert ids
Author: alshahrani2030
Pull request: no

## ❓ Questions & Help
after I Use the BERT tokenizer to convert the tokens to their index numbers in the BERT vocabulary

input_ids = [tokenizer.convert_tokens_to_ids(x) for x in tokenized_texts]

How I can convert them back to the original sentence???

Thank you in advance

Comments:
Hello, you should take a look at the `encode` and `decode` methods in the [documentation](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.decode).

Thank you so much that was very helpful

Glad I could help!

Hi 
I am trying to use the code in this link (https://colab.research.google.com/drive/1pS-eegmUz9EqXJw22VbVIHlHoXjNaYuc#scrollTo=JggjeDC9m2MH) to plot my trained  model
But, I am getting an error. any idea please??
![1](https://user-images.githubusercontent.com/55197626/66767341-2ecd3080-ee7e-11e9-9bc7-4b6f93654793.PNG)
![2](https://user-images.githubusercontent.com/55197626/66767342-2ecd3080-ee7e-11e9-8aac-8743151a95b9.PNG)
Fix compatibility issue with PyTorch 1.2
Author: astariul
Pull request: yes

Using PyTorch 1.2.0 give an error when running XLNet.
We should use the new way to reverse mask : instead of using `1 - mask`, we should use `~mask`

Comments:
Hi,
We can accept this since it breaks lower versions of PyTorch.
You can just feed your mask as a FloatTensor (as indicated in the docstrings I think).
Remove unnecessary use of FusedLayerNorm in XLNet
Author: bryant1410
Pull request: yes

Fix #1172 for XLNet

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1434?src=pr&el=h1) Report
> Merging [#1434](https://codecov.io/gh/huggingface/transformers/pull/1434?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/f3e0218fbb6bcc40b40f10089dae8876654edb23?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1434/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1434?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1434      +/-   ##
==========================================
- Coverage   84.72%   84.72%   -0.01%     
==========================================
  Files          84       84              
  Lines       12591    12590       -1     
==========================================
- Hits        10668    10667       -1     
  Misses       1923     1923
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1434?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/1434/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3hsbmV0LnB5) | `72.09% <100%> (-0.05%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1434?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1434?src=pr&el=footer). Last update [f3e0218...1dea291](https://codecov.io/gh/huggingface/transformers/pull/1434?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Ok, thanks!
Fixed the sample code in the title 'Quick tour'.
Author: e-budur
Pull request: yes

The variable pretrained_weights was fixed to 'bert-base-uncased' to be used in each model to experiment.  Otherwise, the last value of this variable in the previous loop was unintentionally effective in this loop which was causing throwing error.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1505?src=pr&el=h1) Report
> Merging [#1505](https://codecov.io/gh/huggingface/transformers/pull/1505?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a701c9b32126f1e6974d9fcb3a5c3700527d8559?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1505/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1505?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1505   +/-   ##
=======================================
  Coverage   85.98%   85.98%           
=======================================
  Files          91       91           
  Lines       13579    13579           
=======================================
  Hits        11676    11676           
  Misses       1903     1903
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1505?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1505?src=pr&el=footer). Last update [a701c9b...5a8c6e7](https://codecov.io/gh/huggingface/transformers/pull/1505?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks!
Fix import error in script to convert faisreq roberta checkpoints
Author: louismartin
Pull request: yes

Fix ImportError in `convert_roberta_original_pytorch_checkpoint_to_pytorch.py`, see #1459.


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1512?src=pr&el=h1) Report
> Merging [#1512](https://codecov.io/gh/huggingface/transformers/pull/1512?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a701c9b32126f1e6974d9fcb3a5c3700527d8559?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1512/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1512?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1512   +/-   ##
=======================================
  Coverage   85.98%   85.98%           
=======================================
  Files          91       91           
  Lines       13579    13579           
=======================================
  Hits        11676    11676           
  Misses       1903     1903
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1512?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1512?src=pr&el=footer). Last update [a701c9b...49cba6e](https://codecov.io/gh/huggingface/transformers/pull/1512?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Looks good to me, thanks!
Changelog
Author: BramVanroy
Pull request: no

## 🚀 Add changelog between versions

New versions are pushed to PyPi at a steady pace, but it's not evident to find the changes that new versions bring. Is there a changelog anywhere? Something similar to a HISTORY file would be nice. I think it would definitely contribute to better documentation!



Comments:
Hi @BramVanroy, we detail the changes in the ["Releases" section](https://github.com/huggingface/transformers/releases).  Are you thinking of something different?

Having a documentation per-version is on our roadmap, it should help tremendously regarding version changes.

Ah, I was looking inside different major version commits for some sort of changelog file - which still might be useful in itself, as you indicate. But having the Github releases is exactly what I was after! Apologies, should've thought this through.
Plan to support UniLM ?
Author: astariul
Pull request: no

# 🌟New model addition

## Model description

**UniLM** : Pre-trained transformer for sequence to sequence generation.

Paper : https://arxiv.org/pdf/1905.03197.pdf

## Open Source status

* [x] the model implementation is available: **[official Pytorch](https://github.com/microsoft/unilm)**
* [x] the model weights are available: For now only english : [UniLMv1-large-cased](https://github.com/microsoft/unilm#pre-trained-models)

## Additional context

The official implementation is based on a modified version of this repository (version 0.4.0).  
That would be nice to have a unified API :)

*Note : They didn't release the code for pretraining yet.*

Comments:
This is on our mid-term roadmap.
We have a project adding Seq2seq models and UniLM will be part of this project.