noamaanMulla-03/transformers-issues huggingface dataset:
Usage example needs [CLS] and [SEP] added post-tokenization
Author: jplehmann
Pull request: no

Since probably #176, the usage example results in the special tokens getting normalized in a bad way and the assertion clearly fails.
```
['[',
 'cl',
 '##s',
 ']',
 'who',
 'was',
 'jim',
 'henson',
 '[MASK]',
 '[',
 'sep',
 ']',
 'jim',
 'henson',
 'was',
 'a',
 'puppet',
 '##eer',
 '[',
 'sep',
 ']']
```
I believe something like this is intended:
```
text1 = "Who was Jim Henson ?"
text2 = "Jim Henson was a puppeteer"
tokenized_text = ['[CLS]'] + tokenizer.tokenize(text1) + ['[SEP]'] + tokenizer.tokenize(text2) + ['[SEP]']
```

I really appreciate this project!



Comments:
Argh, I just realized that due to dependency conflicts, pip had installed an old version `0.3.0`.

Was fixed here:
https://github.com/huggingface/pytorch-pretrained-BERT/issues/303
Issue#324: warmup linear fixes
Author: lukovnikov
Pull request: yes

Fixes for [Issue#324](https://github.com/huggingface/pytorch-pretrained-BERT/issues/324).

- Using the same schedule functions in BertAdam and OpenAIAdam, fixing `warmup_linear` of OpenAIAdam
- fix for negative learning rate after t_total for `warmup_linear`
- some more docstrings
- warning when t_total is exceeded with `warmup_linear`, implemented inside `.step()` of the optimizer (maybe not that nice). Warning is printed on every batch update.

Comments:
Great, thanks @lukovnikov!
does run_lm_finetuning.py actually use --eval_batch_size?
Author: bsugerman
Pull request: no

I'm looking through this code (thanks so much for writing it, btw) and I'm not seeing whether it actually uses eval_batch_size at all. If it doesn't, is it still performing an evaluation step to assess goodness of fit? 

Comments:
No, there's no evaluation step in the example script yet. What I can recommend is using your downstream task for evaluation of the pretrained BERT. Alternatively, you could of course also add some evaluation of the LM / nextSentence loss on a validation set.

Perhaps for clarity then, that parameter should be taken out of the script? 

Sure, makes sense. I created a PR. Thanks for pointing this out @bsugerman .

Fixed in master now.
Minor redundancy in model defintion?
Author: mttk
Pull request: no

this is a _major_ nitpick but it was a bit confusing at first:

https://github.com/huggingface/pytorch-pretrained-BERT/blob/822915142b2f201c0b01acd7cffe1b05994d2d82/pytorch_pretrained_bert/modeling.py#L206-L212

L212 can simply be replaced by `self.all_head_size = config.hidden_size` as you already error out if the result of division isn't a whole number. 

Comments:
Yes, feel free to submit a PR. Otherwise, I'll fix it in the next release.
RuntimeError: cuda runtime error while running run_classifier.py with 'bert-large-uncased' bert model
Author: laibamehnaz
Pull request: no


Comments:
reduce batch size?

It is 32 as of now . What do you think I should reduce it to ?

Start very low and increase while looking at `nvidia-smi` or a similar GPU memory visualization tool.

Closing this for now, feel free to re-open if you have other issues.
Facing issue in Run Fine tune LM
Author: tuhinjubcse
Pull request: no

So my LM sample.txt is such that each doc has only one line 
So in BERTDataSet len is giving negative 
I tried changing it to self.num_docs - 1
def __len__(self):
        print(self.corpus_lines ,self.num_docs)
        return self.corpus_lines - self.num_docs - 1
I am also getting errors at multiple steps, Is the code written with the assumption that each document will have multiple lines in it?



Comments:
Yes, you need documents with multiple lines because only sentences from the same doc are used as positive examples for the nextSentence prediction. 

Seems like the expected behavior. Feel free to open a PR to extend the example if you want @tuhinjubcse.
Help: how to get index/symbol from last_hidden, on text8?
Author: moonblue333
Pull request: no

I am trying on text8 dataset.  I want to print next token.   The model in source code forward() output is loss, but I want to get logits and softmax result, and finally get next token in vocab.

how to get index/symbol from last_hidden, on text8?

Thanks

Comments:
Hi,
There is no pretrained character-level model for text8 right now.
Only a word-level model trained on wikitext 103.
DataParallel imbalanced memory usage
Author: hongkahjun
Pull request: no

Simialr to this issue: https://discuss.pytorch.org/t/dataparallel-imbalanced-memory-usage/22551/12, when I run  run_lm_finetuning.py using 4 GPUs on Microsoft Azure, the first GPU will have 4000MB Memory usage while the other 3 are at 700MB. The Volatile Util for the first GPU also is at 100% while the rest are at 0%. 

It seems that the solution might be something to do with incorporating the loss calculation in the forward pass but I do not know how to solve it.

Comments:
Managed to get volatile GPU to work properly but memory allocation is sitll imbalanced

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.78       Driver Version: 410.78       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P40           Off  | 00000275:00:00.0 Off |                    0 |
| N/A   43C    P0    61W / 250W |  11151MiB / 22919MiB |     99%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla P40           Off  | 00003984:00:00.0 Off |                    0 |
| N/A   41C    P0    60W / 250W |   5979MiB / 22919MiB |     99%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla P40           Off  | 00005B97:00:00.0 Off |                    0 |
| N/A   38C    P0    63W / 250W |   5979MiB / 22919MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla P40           Off  | 0000EA90:00:00.0 Off |                    0 |
| N/A   42C    P0    61W / 250W |   5979MiB / 22919MiB |     99%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      5927      C   python                                     11141MiB |
|    1      5927      C   python                                      5969MiB |
|    2      5927      C   python                                      5969MiB |
|    3      5927      C   python                                      5969MiB |
+-----------------------------------------------------------------------------+

Yes, there is no mechanism to balance memory in the examples. In NVIDIA's tests, it didn't help.
HugginFace or HuggingFace?
Author: davidefiocco
Pull request: no

Thought to flag, also given the terrific work on this repo (and others), that the company name in the code here seems to be systematically spelt wrong (?)
https://github.com/huggingface/pytorch-pretrained-BERT/search?q=hugginface&unscoped_q=hugginface

Comments:
Thanks, I'll fix that in a future release.
Too much info @ stdout
Author: mahnerak
Pull request: no

As a library, it is preferred to have no unnecessary `print`s in the repo. Using the `pytorch-pretrained-BERT` makes it impossible to use `stdout` as main output mechanism for my code.
For example: it prints directly to `stdout` "Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex.".
I guess printing this kind of messages to `stderr` will be a better idea.

Comments:
Oh that's right, this one should be a logging.info event like the other ones.

Fixed
run_classifier.py :  TypeError: join() argument must be str or bytes, not 'PosixPath'
Author: WilliamTambellini
Pull request: no

when trying the MRPC example : 

python3.5 run_classifier.py \
  --task_name MRPC \
  --do_train \
  --do_eval \
  --do_lower_case \
  --data_dir $GLUE_DIR/MRPC/ \
  --bert_model bert-base-uncased \
  --max_seq_length 128 \
  --train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3.0 \
  --output_dir /tmp/mrpc_output

Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex.
02/22/2019 21:29:48 - INFO - __main__ -   device: cuda n_gpu: 1, distributed training: False, 16-bits training: False
02/22/2019 21:29:48 - INFO - pytorch_pretrained_bert.tokenization -   loading vocabulary file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /home/ubuntu/.pytorch_pretrained_bert/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
02/22/2019 21:29:48 - INFO - __main__ -   LOOKING AT /home/ubuntu/glue_data/MRPC/train.tsv
Traceback (most recent call last):
  File "pytorch-pretrained-BERT/examples/run_classifier.py", line 637, in <module>
    main()
  File "pytorch-pretrained-BERT/examples/run_classifier.py", line 465, in main
    cache_dir = args.cache_dir if args.cache_dir else os.path.join(PYTORCH_PRETRAINED_BERT_CACHE, 'distributed_{}'.format(args.local_rank))
  File "/usr/lib/python3.5/posixpath.py", line 89, in join
    genericpath._check_arg_types('join', a, *p)
  File "/usr/lib/python3.5/genericpath.py", line 143, in _check_arg_types
    (funcname, s.__class__.__name__)) from None
TypeError: join() argument must be str or bytes, not 'PosixPath'


ubuntu 16
python 3.5
torch 1.0.1
Collecting torch>=0.4.1 (from pytorch-pretrained-bert)
  Downloading https://files.pythonhosted.org/packages/59/d2/4e806f73b4b72daab9064c99394fc22ea6ef1fb052154546405057cd192d/torch-1.0.1.post2-cp35-cp35m-manylinux1_x86_64.whl (582.5MB)


Comments:
https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/file_utils.py#L30
`PYTORCH_PRETRAINED_BERT_CACHE = Path(os.getenv('PYTORCH_PRETRAINED_BERT_CACHE',
                                                   Path.home() / '.pytorch_pretrained_bert'))` --> 
`PYTORCH_PRETRAINED_BERT_CACHE = str(Path(os.getenv('PYTORCH_PRETRAINED_BERT_CACHE',
                                                   Path.home() / '.pytorch_pretrained_bert')))` would solve this, I don't know if there are any side effects.

Maybe, a test should be added here?

The above does not resolve the err

Let's rather keep the library's internal using `Path` and fix the examples by adding `str` there instead.
Fixed on master now.
TransfoXLLMHeadModel output interpretation
Author: bergen
Pull request: no

TransfoXLLMHeadModel gives an output of log probabilities of shape [batch_size, sequence_length, n_tokens]. What do these probabilities represent? For example, what distribution is output at the first sequence position? Is it the conditional distribution given the first word? If so, how can the probability of a complete sentence be computed, including the first word? 

Also, the readme states:
> softmax_output: output of the (adaptive) softmax:
>if target is None: Negative log likelihood of shape [batch_size, sequence_length]

This appears to be incorrect. From current behavior, it should say: if target is **not** None

Comments:
Hi,
1/ it's the usual language modeling probabilities: each token probability given the previous tokens
2/ thanks, fixed.
Single sentence corpus in run_lm_finetuning?
Author: vebits
Pull request: no

Hi,

I am trying to pre-train using `BertForPreTraning` in `run_lm_finetuning.py`. My target corpus is based on very many tweets and I am unsure how the model will tackle that since they are mostly only one sentence. Will it affect the IsNextSentence task? 

Should my .txt input file consist of one tweet on each line where each tweet is seperated by an empty line?

Comments:
https://github.com/huggingface/pytorch-pretrained-BERT/issues/272

I had the same issue and but apparently this cant be done in BERT

Yes, can't be done currently. Feel free to submit a PR to extend the `run_lm_finetuning` example @vebits!
run_classifier with evaluation job only
Author: PaulZhangIsing
Pull request: no

Thanks for giving such awesome project.
However, I have encountered some problem.
After training the model, I just want to do eval on another dataset with the trained model. Therefore I only open do_eval.
However, it gives me this error:

Traceback (most recent call last):
  File "run_classifier_torch.py", line 687, in <module>
    main()
  File "run_classifier_torch.py", line 677, in main
    'loss': tr_loss/nb_tr_steps}
UnboundLocalError: local variable 'tr_loss' referenced before assignment

It seems that the loss and tr_loss are only declared in train. If we neglect the training process, the error shall pop up. 

solution : using eval_loss instead

Comments:
 evalonly works last ver, need to mv some code out of train...

> evalonly works last ver, need to mv some code out of train...

yup I agree. I shall do the eval loss and do it


Seems fixed in master, right? Feel free to re-open the issue if it's not the case.
optimizer.zero_grad() in run_openai_gpt.py?
Author: jaminche
Pull request: no

In `run_openai_gpt.py`, should there be a call to `optimizer.zero_grad()` after updating parameters so that we zero out the gradients between minibatches?

https://github.com/huggingface/pytorch-pretrained-BERT/blob/2152bfeae82439600dc5b5deab057a3c4331c62d/examples/run_openai_gpt.py#L212

Comments:
Oh that's a mistake indeed, thanks for pointing out.
Fixed on master.
Fine tuning for evaluation
Author: Alexadar
Pull request: no

Hi!
1) Help me please figure out, what would be optimal batch size for evaluating nextSentencePrediction model? For performance.  Is it same as used during pre-training (128)? 
2) If i building high performance evaluating backend on CUDA, would it be a good idea to use several threads with bert model in each, or its better to use one thread with proper batching?

Comments:
1. For evaluation I would advise the maximum batch size that your GPU allows. You will be able to use more efficiently this way.

2. I think you will be better off by using a single thread.

Thanks! How can i figure out optimal batch size? I want to try tesla k80

You increase it gradually and when the program crashes, it is too big ^^.

Thanks!

Guys, sorry i reopen this issue, but it might be helpful and on topic of evaluation
I want to load batch of data into model for evaluation. Batch have size of 16 sentences of different length
Code:
```
tokens_tensor = torch.tensor(indexed_tokens)
segments_tensors = torch.tensor(segments_ids)
predictions = model(tokens_tensor, segments_tensors)
```
indexed_tokens are array of size 16 of arrays of inputs.
I got error
ValueError: expected sequence of length 121 at dim 1 (got 23)

when i create tensor from a single element
tokens_tensor = torch.tensor([indexed_tokens[0]])
it works

What im doing wrong? 
Thanks!

Could you create of minimal program that reproduces your problem (with the code you are using to generate `indexed_tokens`)?

1. Tensor Input array should have same length for all rows. My sentences had various length. That's why pytorch raise exception
2. If you add zeros to the end of input arrays, to make all rows equal, evaluation will be slower than one per sentence. Batching not improving speed.

Hi @Alexadar, you have to batch your examples and pad them indeed. No other way I'm afraid.

Sorry, i missed your post request for example. 
Yes, padding is only way to batch. It is slower than process sentencess one by one, i tested on GPU.
Tests error: Issue with python3 compatibility, on zope interface implementation
Author: AprilSongRits
Pull request: no
Labels: name: Help wanted

Hi, I came across the following error after run **python -m pytest tests/modeling_test.py**

________________________________________________________________________________ ERROR collecting tests/modeling_test.py __________________________________________________________________________________
modeling_test.py:25: in <module>
    from pytorch_pretrained_bert import (BertConfig, BertModel, BertForMaskedLM,
/usr/local/lib/python3.6/site-packages/pytorch_pretrained_bert/__init__.py:7: in <module>
    from .modeling import (BertConfig, BertModel, BertForPreTraining,
/usr/local/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py:218: in <module>
    from apex.normalization.fused_layer_norm import FusedLayerNorm as BertLayerNorm
/usr/local/lib/python3.6/site-packages/apex/__init__.py:18: in <module>
    from apex.interfaces import (ApexImplementation,
/usr/local/lib/python3.6/site-packages/apex/interfaces.py:10: in <module>
    class ApexImplementation(object):
/usr/local/lib/python3.6/site-packages/apex/interfaces.py:14: in ApexImplementation
    implements(IApex)
/usr/local/lib/python3.6/site-packages/zope/interface/declarations.py:483: in implements
    raise TypeError(_ADVICE_ERROR % 'implementer')
E   TypeError: Class advice impossible in Python3.  Use the @implementer class decorator instead.

**My configurations are as follows:
python version 3.6.4
CUDA Version 8.0.61
torch==1.0.1.post2
apex==0.9.10.dev0
zope.interface==4.6.0**

Thanks

Comments:
any solution here?

This looks like an incompatibility between apex and zope.
Have you tried without installing apex?

> This looks like an incompatibility between apex and zope.
> Have you tried without installing apex?

I uninstalled apex, it works now!
Thank you so much!!!!
How to add input mask to GPT?
Author: jolinxql
Pull request: no

I use `attention_mask` when I do `bert.forward(input, attention_mask)`. But in GPT, when I try to pass a batch of input to `OpenAIGPTModel` to extract a batch of features, and the lengths of sentences in a batch are different, I have no idea how to do it. Or maybe it doesn't need the mask to be given? If so, is zero the padding_index?

For a quick review, this is the code for bert to extract embeddings.
``` python 
all_encoder_layers, pooled_output = self.bert(inputs[:, :seq_max_len], token_type_ids=None,
                                          attention_mask=att_mask.to(device))
embeds = torch.cat(all_encoder_layers[-self.bert_n_layers:],-1)
```

Comments:
GPT is a causal model so each tokens only attend to the left context and masking is not really needed.
Just mask the output according to your lengths (and be such that each input sample start at the very first left token).
Not able to import RandomSampler, Getting error "ImportError: cannot import name 'RandomSampler'"?
Author: Linkyx
Pull request: no

Not able to import RandomSampler, Getting error "ImportError: cannot import name 'RandomSampler'"? Did I get a wrong torch version?

Comments:
how do you fix this issue?

> how do you fix this issue?

try to update your torch version，i found it didn't work in torch 4.0.0, try  "torch >=4.0.1"
Conversion of gpt-2 small model
Author: lahwran
Pull request: no

Hey! This seems like something a lot of folks will want. I'd like to be able to load GPT-2 117M and fine-tune it. What's necessary to convert it? I looked at the tensorflow code a little and it looks vaguely related to transformer xl, but I haven't looked at the paper yet or etc.

Comments:
I'd like to help out on this. I will have a look and try to understand the earlier bridges in this repo. Let me know if you see anywhere a newcomer can be helpful with.

Sure, would be happy to welcome a PR.
You can start from `modeling_openai.py` and `tokenization_openai.py`'s codes.
It's pretty much the same architecture (read OpenAI's paper first!)
You should mostly reimplement the BPE tokenization to work byte-level and move the layer norms modules to the input rather than the output of the layers.

Ok, GPT-2 should be in the coming 0.6.0 release (see #287)

Ok it's on pip: https://github.com/huggingface/pytorch-pretrained-BERT/releases/tag/v0.6.0
Please read the updated README for details. All should be there (model and examples).
Have a nice week y'all.

@thomwolf do we have pytorch implementation of GPT-2 small? 

Yes, just read the README
BertEmbedding not initialized with `padding_idx=0`
Author: haozheji
Pull request: no

https://github.com/huggingface/pytorch-pretrained-BERT/blob/2152bfeae82439600dc5b5deab057a3c4331c62d/pytorch_pretrained_bert/modeling.py#L696

The bert-embeddings are not initialized with `padding_idx=0`, which may potentially result in none zero embeddings for zeros paddings in some early version of pytorch.

Comments:
Could be, do you want to submit a PR to update this?

Closed by #358, thanks @cdjhz!
Make the hyperlink of NVIDIA Apex clickable
Author: bharatr21
Pull request: yes

In the case of the ImportError in modeling.py [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/7cc35c31040d8bdfcadc274c087d6a73c2036210/pytorch_pretrained_bert/modeling.py#L219), make the hyperlink to NVIDIA Apex redirect properly by spacing the '.'

Comments:
Thanks!
handle ImportError exception when used from projects outside
Author: deepbrain
Pull request: yes

The relative path that starts with . does not work when a file is used from an outside project. I added a safe code to handle the ImportError exception in this case, so I can use the source file without having to make local changes to it.

Comments:
Just doing `from pytorch_pretrained_bert import BertTokenizer, BertModel` in your project doesn't work?

That's what they do in [AllenNLP](https://github.com/allenai/allennlp/blob/3f0953d19de3676ea82e642659fc96d90690e34d/allennlp/modules/token_embedders/bert_token_embedder.py#L14) or [flair](https://github.com/zalandoresearch/flair/blob/797c958d0e8c256531f2cea37508e7becb2026cb/flair/embeddings.py#L14)

it does work, however, I don't need the higher level classes for my model, so I am sub classing the OpenAIGPTPreTrainedModel and Block and they are not included into the __init__.py. Including them into the __init__.py also solves the issue with the importException:

from .modeling_openai import (OpenAIGPTConfig, OpenAIGPTModel,
OpenAIGPTLMHeadModel, OpenAIGPTDoubleHeadsModel, OpenAIGPTPreTrainedModel, Block, load_tf_weights_in_openai_gpt)


I see. I think we can fix this the same way I did in the `bert` case by adding `from __future__ import absolute_import`. If you don't mind I'll do a quick PR on that.