noamaanMulla-03/transformers-issues huggingface dataset:
[Question]Embedding Generate Problem
Author: SupUnicorn
Pull request: no
Labels: name: Help wanted

name: Discussion

name: wontfix

When i use 'bert-base-uncase' model to get english embeddings,I can get accurate and unchangeable tensor.But when I do same things on 'bert-base-chinese',I input same sequence every times,but I get different tensors.My input senquece like ''[CLS]+sequence+[SEP]"

Comments:
Hi,

How did you get the english embeddings using the bert-base-uncase model.

I have fine tuned the model on domain specific corpus. I am trying to get embeddings now using the pytorch_model.bin I got after fine tuning.

Any idea on how to do this?

Are your models in evaluation mode (`model.eval()`)?
Usually reproductibility issues comes from forgetting to disable the DropOut modules using `model.eval()`.

> Are your models in evaluation mode (`model.eval()`)?
> Usually reproductibility issues comes from forgetting to disable the DropOut modules using `model.eval()`.

Oh,it's really helpful.My rediculous mistake.Thank you for your answer!!!

@SupUnicorn  Following the above comment, could you please let me know , during the evaluation while reproducing the results using a finetuned model, how to disable the DropOut modules ?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
convert_tf_checkpoint_to_pytorch get different result?
Author: light8lee
Pull request: no
Labels: name: wontfix

I'm using this repo to do Chinese NER on MSRA dataset, when I use the pretrained model `bert-base-chinese ` , the result is very good, it can reach 0.93+ f1 on test set in first epoch. But when I used `convert_tf_checkpoint_to_pytorch` to convert the original bert released checkpoint `chinese_L-12_H-768_A-12` to pytorch_model.bin, the result is very bad, it can only reach 0.5 f1 score on test set. Is there anything I do wrong?
And when I used `BertForPreTraining.from_pretrained`, the result is alright.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
adding vocabulary in OpenAI GPT2 tokenizer issue
Author: Linohong
Pull request: no

Hi, 
I am trying to add few vocabulary tokens to the gpt2 tokenizer
but there seems few problems in adding vocab.

Let's say I want to make sequence like 

> "__bos__" + sequence A + "__seperator__" + sequence B + "__seperator__" + sequence C + "__eos__"

This means that I have to add "__bos__", "__seperator__", "__eos__" tokens to the tokenizer.
I've found 
 token already in the vocab list, but I wanted to use those special 
symbols to reflect my special intentions to treat input sequence.
However, when I succesfully added tokens to the vocab list by fixing some of the codes 
in the 'tokenization_utils.py' file just like below, 


```
# mark this line of code as a comment
# if self.convert_tokens_to_ids(token) == self.convert_tokens_to_ids(self.unk_token):
```


it works fine at the training stage, but the index mapping went totally different in the 
evaluation phase. 

Should I use random but unused token which already is in the vocab list of the tokenizer
to replace my special tokens? 
For example, if there were some random "^&*" token exist in the vocab list,
use that token as my __bos__ token instead.


Anyway, thank you for providing such a legendary libraries opened! 
Thank you very much :) 

Comments:
What specifically did you change in `tokenization_utils.py`? 

> it works fine at the training stage, but the index mapping went totally different in the
> evaluation phase.

Can you elaborate on what you mean? Perhaps post some output? Is it hanging? Or are you just getting wildly poor performance once you move to eval?




@brendanxwhitaker 
thanks for asking !! :)
I found solution thanks to #799 , 
the problem was solved by adding 

`model.resize_token_embeddings(len(tokenizer))`

line when recalling my model !
The problem was that I skipped over the part 
where I had to resize the scale of the vocab to that of 
when I add new tokens. 

Thank you ! :)
git pull pytorch-transformers??
Author: Linohong
Pull request: no

Hello, 
I have git cloned 'pytorch-pretrained-bert' before there is a new release, pytorch-transformers
and I added many of the comments and new example files in the cloned project. 
However, when I found there has been a new version released, git pulling didn't work 
for conflicting files issues. 
Is it because of the new release of 'pytorch-transformers' conflicts to the older version which 
is totally different in name? 
 

Comments:
I suspect this is more of a general `git` question. We'll close this unless there is something specific to the lib.

Good luck!

@julien-c ,
yeah, I had some of code mismatch issues and that falls into general github matters 
and I got it done :) 

Thank you  :)
Is loss.mean() needed?
Author: ZhaofengWu
Pull request: no
Labels: name: wontfix

In `run_classifier.py`, there is a:
https://github.com/huggingface/pytorch-pretrained-BERT/blob/3fc63f126ddf883ba9659f13ec046c3639db7b7e/examples/run_classifier.py#L841-L842

However, a couple of lines higher, the logits are already flattened
https://github.com/huggingface/pytorch-pretrained-BERT/blob/3fc63f126ddf883ba9659f13ec046c3639db7b7e/examples/run_classifier.py#L834-L839

So I assume the loss returned will only be a single-element tensor, correct? I also tested this code on 2 gpus and there indeed is only one single-element tensor.

Comments:
This line is used when people use multi-gpu in a single python process (parallel instead of distributed). This is not the recommended setting (distributed is usually faster).

I wrote a blog post on this (parallel/distributed and the like) a few months ago: https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255

So on a multi-GPU machine, if we run `run_classifier.py`, by default it uses the distributed setting, correct? I'm assuming the answer is yes because when I ran this code on a 2 gpu machine, the `loss` was already a single-element tensor before the `.mean()` call.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Mixed up isNextSentence label in simple_lm_finetuning.py script? 
Author: yakazimir
Pull request: no
Labels: name: Need more information

name: Discussion

I'm wondering if the isNextsentence "label" in the below function is correct? Shouldn't the label be 1 in the case that t1,t2 are taken from self.get_corpus_line(index) (i.e., the first condition on line 150), and 0 if it is random (line 153)? 

https://github.com/huggingface/pytorch-pretrained-BERT/blob/c36cca075a32f59a5ec2083e1d39e7d6564c105b/examples/lm_finetuning/simple_lm_finetuning.py#L141-L157

Comments:
Hi, why should it be the other way around?

I think I mixed up the meaning of 0 and 1 in this context and maybe wrote this post a bit too quickly before looking deeper into the code and documentation.. (sorry!). On second glance, the documentation for the BertForPreTraining is rather clear: 

https://github.com/huggingface/pytorch-pretrained-BERT/blob/d76a57b0ba198eee27b3777f57fcabb6aba8b965/pytorch_pretrained_bert/modeling.py#L766 

I was confused why 0 should mean "true" is this case (i.e., is a next sentence continuation) since in classification 0 often means "false", but whatever, the way it is written is sound (albeit a little counterintuitive at first glance).  

@yakazimir yeah I was confused too. Thanks for your research
run_squad.py F1 and EM score are differ from tensorflow version 
Author: rkcalnode
Pull request: no
Labels: name: wontfix

I use squad1.1 data from https://www.kaggle.com/stanfordu/stanford-question-answering-dataset

I ran convert_tf_checkpoint_to_pytorch.py (using google's uncased_L-12_H-768_A-12 model) and run_squad.py. result was
```
{"exact_match": 73.30179754020814, "f1": 82.10116863001393}
```

Alse I ran run_squad_hvd.py from https://github.com/lambdal/bert (it wraps original tensorflow BERT with horovod) and result was
```
{"exact_match": 80.66225165562913, "f1": 88.09365604437863}
```

bert_config.json is the same.

my parameter's are

pytorch ver
```
--train_batch_size", default=32,
--learning_rate", default=5e-5.
--max_seq_length", default=384,
--doc_stride", default=128,
--max_query_length", default=64
--predict_batch_size",default=8
```

tensorflow ver
```
--train_batch_size=8,
--learning_rate=5e-5,
--num_train_epochs=3.0.
--max_seq_length=384,
--doc_stride=128,
--max_query_length=64,
--predict_batch_size=8
```

I wonder
1. If src pretrained model and parameters are the same, results of pytorch version and tensorflow version are too near score, perhaps F1:88, EM:80. Is it right?
2. If 1. is true, my parameter or procedure is wrong.Where should I see?


Comments:
I also can't get the same result on my squad dataset

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Isn't it too few activations?
Author: mzmey37
Pull request: no
Labels: name: wontfix

Hello, I have a question. In BertLayer part of model we see, that in BertAttention module we do attention (nonlinear action) and selfOutput (linear transformation as it is dense + BN). Then we do BertIntermediate, starting with linear transformation(which means, we have dense, BN, dense transformations going one by one). Why don't we have nonlinear activation between BertAttention and BertIntermediate in BertLayer?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Feature request: roBERTa
Author: chenyangh
Pull request: no

Hi, thanks for making a unified framework for all transformer-based models. Just out of curiosity, do you plan to add the roBERTa pre-trained models? Although FairSeq has provided the model, I still prefer using your framework. Thanks again, big fan. 🤗🤗🤗

Comments:
See #829 (and thanks for the kind words!)
run_glue : Evaluating in every grad_accumulation_step if flag eval during training is true
Author: pratyay-banerjee
Pull request: no

https://github.com/huggingface/pytorch-transformers/blob/f2a3eb987e1fc2c85320fc3849c67811f5736b50/examples/run_glue.py#L154

Comments:
```
if (step + 1) % args.gradient_accumulation_steps == 0:
                scheduler.step()  # Update learning rate schedule
                optimizer.step()
                model.zero_grad()
                global_step += 1

                if args.local_rank in [-1, 0] and args.logging_steps > 0 and global_step % args.logging_steps == 0:
                    # Log metrics
                    if args.local_rank == -1 and args.evaluate_during_training:  # Only evaluate when single GPU otherwise metrics may not average well
                        results = evaluate(args, model, tokenizer)
                        for key, value in results.items():
                            tb_writer.add_scalar('eval_{}'.format(key), value, global_step)
```


Found the fix.
fp16 is still has the problem
Author: zsk423200
Pull request: no

hello, as mentioned in #868 and #871 ,fp16 is broken, and you have fixed in the master once, but i am afraid it also has problem, DP is need after the amp.initialize() too.  i review the code of apex, found that amp do not support the model of parallel type:
```
def check_models(models):
    for model in models:
        parallel_type = None
        if isinstance(model, torch.nn.parallel.DistributedDataParallel):
            parallel_type = "torch.nn.parallel.DistributedDataParallel"
        if isinstance(model, apex_DDP):
            parallel_type = "apex.parallel.DistributedDataParallel"
        if isinstance(model, torch.nn.parallel.DataParallel):
            parallel_type = "torch.nn.parallel.DataParallel"
        if parallel_type is not None:
            raise RuntimeError("Incoming model is an instance of {}. ".format(parallel_type) +
                "Parallel wrappers should only be applied to the model(s) AFTER \n"
                "the model(s) have been returned from amp.initialize.")
```

and the other question is that after i fixed the problem and can run with fp16, but i found it speed the same time and gpu memory, why?


Comments:
yes, I fixed it in #896 and waiting for author to merge..
but I still can't figure out why fp16 didn't save memory and didn't speed up....

Merged

close
Torchscipt mode for BertForPreTraining
Author: loopdigga96
Pull request: no

Hello, i used code from this tutorial https://huggingface.co/pytorch-transformers/torchscript.html
pytorch-transformers==1.0.0

```
from pytorch_pretrained_bert import BertModel, BertTokenizer, BertConfig
import torch

enc = BertTokenizer.from_pretrained("bert-base-uncased")

# Tokenizing input text
text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = enc.tokenize(text)

# Masking one of the input tokens
masked_index = 8
tokenized_text[masked_index] = '[MASK]'
indexed_tokens = enc.convert_tokens_to_ids(tokenized_text)
segments_ids = [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]

# Creating a dummy input
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])
dummy_input = [tokens_tensor, segments_tensors]

# Initializing the model with the torchscript flag
# Flag set to True even though it is not necessary as this model does not have an LM Head.
config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768,
    num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072, torchscript=True)

# Instantiating the model
model = BertModel(config)

# The model needs to be in evaluation mode
model.eval()

# Creating the trace
traced_model = torch.jit.trace(model, [tokens_tensor, segments_tensors])
torch.jit.save(traced_model, "traced_bert.pt")
```

And then I get an error:

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-10-d850721ce4cc> in <module>
     22 # Flag set to True even though it is not necessary as this model does not have an LM Head.
     23 config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768,
---> 24     num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072, torchscript=True)
     25 
     26 # Instantiating the model

TypeError: __init__() got an unexpected keyword argument 'torchscript'
```



Then I looked at source code and did not find torchscript argument in constructor. Help me please?

Comments:
I realised what was the problem. There is an error in documentation.
```from pytorch_pretrained_bert import BertModel, BertTokenizer, BertConfig```
should be
```from pytorch_transformers import BertModel, BertTokenizer, BertConfig```

And also [there](https://github.com/huggingface/pytorch-transformers/blob/master/docs/source/torchscript.rst) is fixed version, but webview documentation looks outdated.

Indeed. We will update the web documentation, thanks for the report
link to `swift-coreml-transformers`
Author: julien-c
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/933?src=pr&el=h1) Report
> Merging [#933](https://codecov.io/gh/huggingface/pytorch-transformers/pull/933?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/f2a3eb987e1fc2c85320fc3849c67811f5736b50?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/933/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/933?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master     #933   +/-   ##
=======================================
  Coverage   79.04%   79.04%           
=======================================
  Files          34       34           
  Lines        6242     6242           
=======================================
  Hits         4934     4934           
  Misses       1308     1308
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/933?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/933?src=pr&el=footer). Last update [f2a3eb9...200da37](https://codecov.io/gh/huggingface/pytorch-transformers/pull/933?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
In run_classifier.py, is "warmup_proportion" a fraction or percentage?
Author: neurite
Pull request: no
Labels: name: wontfix

In `run_classifier.py`, the arg parameter `--warmup_proportion` [help doc](https://github.com/huggingface/pytorch-pretrained-BERT/blob/3fc63f126ddf883ba9659f13ec046c3639db7b7e/examples/run_classifier.py#L628) says, "E.g., 0.1 = 10%% of training.".  Is it actually a percentage such that `0.1` => `0.1%` => `0.001`, which is indeed `10%%` as stated in the help doc?  But throughout the code, it seems `0.1` is just fraction which is `0.1`, not `0.001`. Please clarify and fix the help doc if it is wrong.

Comments:
Found the same problem. `0.1` means `10%` in [Google's TensorFlow implementation](https://github.com/google-research/bert/blob/d66a146741588fb208450bde15aa7db143baaa69/run_classifier.py#L92).

It's a fraction of total training like indicated in the help doc: `0.1 = 10% of training`

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
BertModel docstring missing pooled_output
Author: sleepinyourhat
Pull request: no

The BERT docstring describes three outputs here:

https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/modeling_bert.py#L626

But none of these correspond to the pooled_output output that's added here:

https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/modeling_bert.py#L713

I may be missing something, but this looks like a dated docstring.

Comments:
Damned, missed that one, you are right.
Adding the missing doc-string:
```
**pooler_output**: ``torch.FloatTensor`` of shape ``(batch_size, hidden_size)``
    Last layer hidden-state of the first token of the sequence (classification token)
    further processed by a Linear layer and a Tanh activation function. The Linear
    layer weights are trained from the next sentence prediction (classification)
    objective during Bert pretraining. This output is usually *not* a good summary
    of the semantic content of the input, you're often better with averaging or pooling
    the sequence of hidden-states for the whole input sequence.
```
We'll probably do a small release in a few days once we have gathered all the feedbacks from the main release. In the meantime, I'll set up PyTorch-Hub so people can get the models from master.

A minor edit with the final two optional outputs `hidden_states` and `attentions` are tuples, not lists.

cc @LysandreJik :)

The documentation is outdated regarding that issue. It should probably be re-compiled :-)
extract_features
Author: zhangatao
Pull request: no
Labels: name: wontfix

Traceback (most recent call last):
  File "extract_features.py", line 297, in <module>
    main()
  File "extract_features.py", line 230, in main
    tokenizer = BertTokenizer.from_pretrained(args.bert_model, do_lower_case=args.do_lower_case)
  File "/home/py36/lib/python3.6/site-packages/pytorch_pretrained_bert/tokenization.py", line 197, in from_pretrained
    tokenizer = cls(resolved_vocab_file, *inputs, **kwargs)
  File "/home/py36/lib/python3.6/site-packages/pytorch_pretrained_bert/tokenization.py", line 97, in __init__
    self.vocab = load_vocab(vocab_file)
  File "/home/py36/lib/python3.6/site-packages/pytorch_pretrained_bert/tokenization.py", line 56, in load_vocab
    token = reader.readline()
  File "/home/py36/lib/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte



Hello, the pre-trained model I downloaded from https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased.tar.gz, what caused the above error?

Comments:
Were you able to fix this issue? If yes, can you please share your solution?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
AttributeError: 'BertModel' object has no attribute '_load_from_state_dict'
Author: Shandilya21
Pull request: no

Hi,
I am getting these error, even the i tried the model straight from the repository examples for the test cases.? anyone help me to understand this issues 

Thanks for help in advance


Comments:
You are probably not using the new release of PyTorch-transformers.
Try `pip install pytorch-transformers --upgrade`.
And read the full [readme](https://github.com/huggingface/pytorch-transformers), there are several breaking changes.

@thomwolf Thanks for your update, it works for me !!
BertAdam's get_lr() not return correct learning rate
Author: light8lee
Pull request: no
Labels: name: wontfix

when the initial values of `BertAdam`'s `params` have `requires_grad=False` Parameter, it will just continue the loop in `step()` function line 251, after step() function, when I want to use `get_lr()` to get the current learning rate, the state of this Parameter is a empty dict, so the function just return `[0]` in line 230. I think it is out of expectation, maybe more checking of the grad is needed somewhere.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.