noamaanMulla-03/transformers-issues huggingface dataset:
Fix convert to tf
Author: dhpollack
Pull request: yes

I struggled with this same problem for a long time.  The naive `assign` op way, puts all of the weights into both the checkpoint file (`.ckpt.data-XXXXX-of-YYYYY`) and the meta file (`.ckpt.meta`).  This is because assign adds an operation to the graph.  So basically, you have two instructions in your meta file, one that initializes the variable with random values and then another that assigns the pytorch values to these tensors.  But really, you want to initialize everything once with the meta file and then read the data file which should have your pytorch weights in it.  Tensorflow hides this functionality deep within it's source code and every answer on stackoverflow tells one to use `assign`.  But the `tf.keras.backend.set_value` function does simply replace the weights of a variable.  However, this function makes some assumptions about your session and your graph so I had to change your code a bit.  Long story short, doing easy things in tensorflow is hard.

So what's the difference?

1. the meta file will be about 1mb instead of 400+ mb
2. the script runs in about 10 seconds instead of 3 minutes



 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/907?src=pr&el=h1) Report
> Merging [#907](https://codecov.io/gh/huggingface/pytorch-transformers/pull/907?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/46cc9dd2b51a152b2e262ec12e40dddd13235aba?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/907/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/907?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master     #907   +/-   ##
=======================================
  Coverage   79.03%   79.03%           
=======================================
  Files          34       34           
  Lines        6234     6234           
=======================================
  Hits         4927     4927           
  Misses       1307     1307
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/907?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/907?src=pr&el=footer). Last update [46cc9dd...09ecf22](https://codecov.io/gh/huggingface/pytorch-transformers/pull/907?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Ok thanks David!
Export to Tensorflow not properly implemented
Author: dhpollack
Pull request: no

Apologies for going about this backwards.  I created a pull request #907 to fix your implementation of converting pytorch weights to tensorflow weights.  As explained in the PR, the current implementation puts the weights from the pytorch model into two places in the newly created tensorflow checkpoint.  The fix not only reduces the size of the meta file, but also reduces the running time.  

Comments:
Thanks @dhpollack !
Code snippet on docs page using old import
Author: andrewnc
Pull request: no

This is a documentation issue. I couldn't find where to edit the website source https://huggingface.co/pytorch-transformers/torchscript.html

On that page the code snippet still uses `from pytorch_pretrained_bert import BertModel, BertTokenizer, BertConfig`

The documentation in this repo under [https://github.com/huggingface/pytorch-transformers/blob/master/docs/source/torchscript.rst](url) is correct. 

This seems like a simple sync error

Comments:
Thanks!
Feature Request : run_swag with XLNet and XLM 
Author: pratyay-banerjee
Pull request: no

It would be great if the run_swag script too was updated with XLNet and XLM models. They should be similar to BertForMultipleChoice ?

Comments:
Yes, don't have bandwith for that in the short term. If you want to give it a go feel free.
Closing this issue in favor of the previous one #931
RestructuredText table for pretrained models.
Author: LysandreJik
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/791?src=pr&el=h1) Report
> Merging [#791](https://codecov.io/gh/huggingface/pytorch-transformers/pull/791?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/b33a385091de604afb566155ec03329b84c96926?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/791/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/791?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master     #791   +/-   ##
=======================================
  Coverage   78.91%   78.91%           
=======================================
  Files          34       34           
  Lines        6193     6193           
=======================================
  Hits         4887     4887           
  Misses       1306     1306
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/791?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/791?src=pr&el=footer). Last update [b33a385...9d381e7](https://codecov.io/gh/huggingface/pytorch-transformers/pull/791?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
<model>ForQuestionAnswering loading non-deterministic weights
Author: aychang95
Pull request: no

I was comparing the weight and bias parameters of two different pre-trained-loaded BertForQuestionAnswering model, and they seem to differ.  This causes every instantiation of pre-trained models to have slightly different results.

Compared to #695 where you set the model to eval mode to deactivate dropout layers, the non-deterministic trait seems to come from loading pre-trained models with `BertForQuestionAnswering.from_pretrained("bert-base-uncased")`

To replicate what I'm talking about, you can see below.
```python
model_1 = BertForQuestionAnswering.from_pretrained("bert-base-uncased")
model_2 = BertForQuestionAnswering.from_pretrained("bert-base-uncased")

weights_1 = model_1.state_dict()['qa_outputs.weight']
weights_2 = model_2.state_dict()['qa_outputs.weight']

torch.eq(weights_1, weights_2)

```
This also occurs in XLNetForQuestionAnswering and I was curious to how/why this works that way?

Comments:
These weights are not pretrained, they are added for fine-tuning the model on a downstream question answering task. You have to train the `qa_output` weights.
They are initialized randomly and so will be different at each run.
CONFIG_NAME and WEIGHTS_NAME are missing in modeling_transfo_xl.py
Author: tomohideshibata
Pull request: no

When I run `convert_transfo_xl_checkpoint_to_pytorch.py`, the following error occurs.
```
Traceback (most recent call last):
  File "convert_transfo_xl_checkpoint_to_pytorch.py", line 27, in <module>
    from pytorch_transformers.modeling_transfo_xl import (CONFIG_NAME,
ImportError: cannot import name 'CONFIG_NAME'
```

So, in `modeling_transfo_xl.py`, `from .modeling_utils import (PretrainedConfig, PreTrainedModel, add_start_docstrings)` should be `from .modeling_utils import (CONFIG_NAME, WEIGHTS_NAME, PretrainedConfig, PreTrainedModel, add_start_docstrings)`.

Comments:
Thanks!
How to train BertModel
Author: akshayudnur
Pull request: no

Hi,
I am trying to train BertModel on my domain-based dataset. Please let me know how to train the BertModel.

Comments:
Hi, there are examples in the "examples" folder on finetuning language models. Please take a look at [the scripts available here](https://github.com/huggingface/pytorch-transformers/tree/master/examples/lm_finetuning).
Missing lines in Readme examples?
Author: nicolas-ivanov
Pull request: no

1. In the [example](https://github.com/huggingface/pytorch-transformers#serialization)
```
...
### Do some stuff to our model and tokenizer
# Ex: add new tokens to the vocabulary and embeddings of our model
tokenizer.add_tokens(['[SPECIAL_TOKEN_1]', '[SPECIAL_TOKEN_2]'])
model.resize_token_embeddings(len(tokenizer))
# Train our model
train(model)
...
```

`model.train()` is missing before `train(model)` ?

2. In the [example](https://github.com/huggingface/pytorch-transformers#optimizers-bertadam--openaiadam-are-now-adamw-schedules-are-standard-pytorch-schedules)
```
...
### In PyTorch-Transformers, optimizer and schedules are splitted and instantiated like this:
optimizer = AdamW(model.parameters(), lr=lr, correct_bias=False)  # To reproduce BertAdam specific behavior set correct_bias=False
scheduler = WarmupLinearSchedule(optimizer, warmup_steps=num_warmup_steps, t_total=num_total_steps)  # PyTorch scheduler
### and used like this:
for batch in train_data:
    loss = model(batch)
    loss.backward()
    scheduler.step()
    optimizer.step()
```

`optimizer.zero_grad()` is missing after `optimizer.step()` ?

Comments:
Thanks

@thomwolf please, note: the first example hasn't been fixed by the commit.

Yes, doesn't look like a problem to me. Usually, people put the model in training mode inside the train function (and even inside the training loop I would recommend).

Ok, got it!
RuntimeError: bool value of Tensor with more than one value is ambiguous
Author: letsgoduke
Pull request: no

## ❓ Questions & Help

<!-- Using tokenizer to decode tensor is throwing this error: RuntimeError: bool value of Tensor with more than one value is ambiguous -->

Here's the code I'm trying to run, the tensor itself gets returned, but when I try to decode it I get the error above.  

Any ideas?  Thanks! 

`if __name__ == '__main__':
	# main()
	model_class, tokenizer_class = MODEL_CLASSES['gpt2']
	tokenizer = tokenizer_class.from_pretrained('gpt2')
	context_tokens = tokenizer.encode("My favorite first date idea is")

	model = model_class.from_pretrained('gpt2')
	model.to('cpu')
	model.eval()

	out = sample_sequence(
		model=model,
		context=context_tokens,
		length=140,
		temperature=0.9,
		top_k=1,
		top_p=0.9
	)

	text = tokenizer.decode(out)
	print(text)`



Comments:
Could you please provide more information, especially regarding the `sample_sequence` function and where it is coming from?

Thanks for the response!

My goal is to wrap the GPT2 model interface in a function so I can input a prompt and output generated text.  I'm trying to adapt one of the examples, and I'm getting there, but I wasn't able to find anything on the error specific to Pytorch-Transformers.

Here's the `sample_sequence` function:

    def sample_sequence(model, length, context, num_samples=1, temperature=1, top_k=0, top_p=0.0, device='cpu'):

	context = torch.tensor(context, dtype=torch.long, device=device)
	context = context.unsqueeze(0).repeat(num_samples, 1)
	generated = context
	with torch.no_grad():
		for _ in trange(length):
			inputs = {'input_ids': generated}
			outputs = model(**inputs)
			next_token_logits = outputs[0][0, -1, :] / temperature
			filtered_logits = top_k_top_p_filtering(next_token_logits, top_k=top_k, top_p=top_p)
			next_token = torch.multinomial(F.softmax(filtered_logits, dim=-1), num_samples=1)
			generated = torch.cat((generated, next_token.unsqueeze(0)), dim=1)
		return generated

Your sample sequence function returns a `torch.tensor([[int64]])`, of shape `[batch_size, sequence_length]`. In your specific case it is of size `[1, 146]`.

You cannot feed such an object to the tokenizer for decoding, as it only accepts a list of integers.

You can fetch the list of integers by calling the `tolist()` method on your output, and then feed it to the tokenizer for decoding:

```
out = sample_sequence(
	model=model,
	context=context_tokens,
	length=140,
	temperature=0.9,
	top_k=1,
	top_p=0.9
)

generated_list = out[0].tolist()
text = tokenizer.decode(generated_list)

print(text)
```

It worked!

This was very helpful, thank you!  Just getting familiar with this package, it's awesome!
use of special tokens in gpt2?
Author: 2016csb1062
Pull request: no
Labels: name: wontfix

![Screenshot from 2019-05-31 23-36-39](https://user-images.githubusercontent.com/36623119/58725591-24455980-83fd-11e9-81c6-d35fa4370e8e.png)
i am actually new to this field , I am trying to use gpt2 for sequence classification task in which i am adding "
" after each sequence and using  last hidden state for classification.  My doubt is what is the use of special tokens such as "
" if the gpt2tokenizer does not recognize them?
thank you and any suggestion in classification task is appreciated!
 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Fixed typo in migration guide
Author: chrisgzf
Pull request: yes

This PR fixes a minor typo in the migration guide. `weights` was misspelled as `weigths`

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/977?src=pr&el=h1) Report
> Merging [#977](https://codecov.io/gh/huggingface/pytorch-transformers/pull/977?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/4fc9f9ef54e2ab250042c55b55a2e3c097858cb7?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/977/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/977?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master     #977   +/-   ##
=======================================
  Coverage   79.16%   79.16%           
=======================================
  Files          38       38           
  Lines        6406     6406           
=======================================
  Hits         5071     5071           
  Misses       1335     1335
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/977?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/977?src=pr&el=footer). Last update [4fc9f9e...a6f412d](https://codecov.io/gh/huggingface/pytorch-transformers/pull/977?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Ok, thanks!
Fixing unused weight_decay argument
Author: ethanjperez
Pull request: yes

Currently the L2 regularization is hard-coded to "0.01", even though there is a --weight_decay flag implemented (that is unused). I'm making this flag control the weight decay used for fine-tuning in this script.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/960?src=pr&el=h1) Report
> Merging [#960](https://codecov.io/gh/huggingface/pytorch-transformers/pull/960?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/44dd941efb602433b7edc29612cbdd0a03bf14dc?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/960/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/960?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master     #960   +/-   ##
=======================================
  Coverage   79.04%   79.04%           
=======================================
  Files          34       34           
  Lines        6242     6242           
=======================================
  Hits         4934     4934           
  Misses       1308     1308
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/960?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/960?src=pr&el=footer). Last update [44dd941...28ba345](https://codecov.io/gh/huggingface/pytorch-transformers/pull/960?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Indeed, thanks Ethan!
Fixed small typo
Author: saket404
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/958?src=pr&el=h1) Report
> Merging [#958](https://codecov.io/gh/huggingface/pytorch-transformers/pull/958?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/44dd941efb602433b7edc29612cbdd0a03bf14dc?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/958/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/958?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master     #958   +/-   ##
=======================================
  Coverage   79.04%   79.04%           
=======================================
  Files          34       34           
  Lines        6242     6242           
=======================================
  Hits         4934     4934           
  Misses       1308     1308
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/958?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/958?src=pr&el=footer). Last update [44dd941...836e513](https://codecov.io/gh/huggingface/pytorch-transformers/pull/958?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Nice!
Fix comment typo
Author: FeiWang96
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/955?src=pr&el=h1) Report
> Merging [#955](https://codecov.io/gh/huggingface/pytorch-transformers/pull/955?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/44dd941efb602433b7edc29612cbdd0a03bf14dc?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/955/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/955?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master     #955   +/-   ##
=======================================
  Coverage   79.04%   79.04%           
=======================================
  Files          34       34           
  Lines        6242     6242           
=======================================
  Hits         4934     4934           
  Misses       1308     1308
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/955?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/955/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvbW9kZWxpbmdfYmVydC5weQ==) | `87.98% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/955?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/955?src=pr&el=footer). Last update [44dd941...a24f830](https://codecov.io/gh/huggingface/pytorch-transformers/pull/955?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks!
The pre-trained model you are loading is a cased model but you have not set `do_lower_case` to False.
Author: loretoparisi
Pull request: no

I initialized the tokenizer and the model like

```python
def load_bert_score_model(bert="bert-base-multilingual-cased", num_layers=8):
    
    assert bert in bert_types

    tokenizer = BertTokenizer.from_pretrained(bert, do_lower_case=True)
    model = BertModel.from_pretrained(bert)
    model.eval()
    device = 'cuda' if torch.cuda.is_available() else 'cpu'
    model.to(device)

    # drop unused layers
    model.encoder.layer = torch.nn.ModuleList([layer for layer in model.encoder.layer[:num_layers]])

    return model, tokenizer
```

so setting the `do_lower_case=True`, but I'm getting this warning:

```
The pre-trained model you are loading is a cased model but you have not set `do_lower_case` to False. We are setting `do_lower_case=False` for you but you may want to check this behavior.
```

Comments:
Hi! You seem to be loading a cased model (such as the `bert-base-multilingual-cased`), but you're specifying `do_lower_case` to your tokenizer, which strips accents and lowercases every character.

The model you specified has been trained with uppercase and lowercase characters as well as accent markers, so you should use it with such characters as well. If you're looking at using only lowercase characters, it would be better for you to use an uncased model (such as the `bert-base-multilingual-uncased`).

@LysandreJik that is correct, thank you.
total training steps and tokenization in run_glue
Author: xinsu626
Pull request: no

Question for total Training Steps: 

In run_glue line 78, the total number of training steps is calculated using `t_total = len(train_dataloader) // args.gradient_accumulation_steps * args.num_train_epochs`.

I was wondering if we use gradient accumulation the num_train_epochs in the above code is not actual training epochs instead the actual training epochs may be `args.num_train_epochs/args.gradient_accumulation_steps`. So the total training steps should be `t_total = (len(train_dataloader) // args.gradient_accumulation_steps) * (args.num_train_epochs / args.gradient_accumulation_steps)`. Is my understanding correct?

Question for tokenization:

I saw in your `utils_glue.py`'s `convert_examples_to_features` function, you set `cls_token_at_end=False, pad_on_left=False` , but didn't provide any accesses to change these parameters when users want to fine tune xlnet. Does this will decrease the xlnet fine tuning performance? 

Thank you. 

Comments:
How to output a vector
Author: Wenenen
Pull request: no

How to use BertModel to output a word's vector which like a vector in word2vec?

Comments:
Hi, you can use the BertModel to give you the encoded representation of the word ids you have as input. The tensor output by the model’s last layer (of dimension `(batch_size, sequence_length, 768)` for the BertModel) can be considered as the BERT-encoded representation of your input and then be used as input for a downstream task. Is this what you were looking for?

I want to get the word embedding.
Is the following code correct?
    model = BertModel.from_pretrained('ms')
    embedding = model.embeddings.word_embeddings
‘ms’ is my pretrained bert model path

Yes, that works!

thanks!