noamaanMulla-03/transformers-issues huggingface dataset:
understanding of the output from TransfoXLModel 
Author: cherepanovic
Pull request: no
Labels: name: wontfix

the output of the TransfoXLModel has the size of [1, 3, 1024] if the Input has tree tokens. 

`predictions, mems = model(tokens_tensor, mems=None)`

doc from code is 

```
    Outputs:
        A tuple of (last_hidden_state, new_mems)
        `last_hidden_state`: the encoded-hidden-states at the top of the model
            as a torch.FloatTensor of size [batch_size, sequence_length, self.config.d_model]
        `new_mems`: list (num layers) of updated mem states at the entry of each layer
            each mem state is a torch.FloatTensor of size [self.config.mem_len, batch_size, self.config.d_model]
            Note that the first two dimensions are transposed in `mems` with regards to `input_ids` and `target`
```

could the output be explained more precisely.  I would be very grateful! 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
key error when using run_classifier.py in predict mode, expecting label?
Author: search4mahesh
Pull request: no
Labels: name: wontfix

Hi,

I am getting key error when using run_classifier.py in predict mode.

https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_classifier.py

At prediction time we don't have labels hence it gives key error.
run_squad example is good as it was having is_training flag.
Could you please suggest?

if output_mode == "classification":
            label_id = label_map[example.label]
        elif output_mode == "regression":
            label_id = float(example.label)
        else:
            raise KeyError(output_mode)

Thanks
Mahesh

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
BertAdam gradient clipping is not global
Author: raulpuric
Pull request: no
Labels: name: wontfix

Just took a look at the gradient clipping algorithm used in: https://github.com/huggingface/pytorch-pretrained-BERT/blob/3ae8c8be1e3fc770968cd3fdb3b643e0b166e540/pytorch_pretrained_bert/optimization.py#L270

It's clipping gradients to a local norm of 1. It should be clipping gradients to a global norm of 1 as in https://github.com/google-research/bert/blob/master/optimization.py#L74 or  in https://github.com/NVIDIA/Megatron-LM/blob/master/pretrain_bert.py#L226 .

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
The number of train examples in STS-B is only 5749
Author: Dawn90
Pull request: no
Labels: name: wontfix

Hi,

Thanks a lot for the amazing work!
Here's my issue:
When I run the './example/run_classification.py' with task STS-B, I found the train example number is only 5749, less than 7k which was reported in the paper ([paper link](https://www.nyu.edu/projects/bowman/glue.pdf)).

Thanks again!
Best,
Dong

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
how to get the word vector from bert pretrain model ?
Author: zhangyu68
Pull request: no

Could you please help me?
I just want to get bert's word vector,but I only can get the encoder's result. How can I get the word vector before data inputing the encoder model ?
Thank you ! 

Comments:
This will be possible in the new release out soon.

I find a method that can get the words embeddings.Thank you all the same!
self.model = BertModel.from_pretrained(config.bert_path)
self.word_emb = self.model.embeddings
BERT + PyTorch + XLA
Author: snakers4
Pull request: no
Labels: name: wontfix

Hi,

Many thanks for your amazing library!
Even though no models were shared for Russian, we used your interfaces with success when doing some [research](https://towardsdatascience.com/complexity-generalization-computational-cost-in-nlp-modeling-of-morphologically-rich-languages-7fa2c0b45909).

Anyway here is my question.
Did you try [this](https://github.com/pytorch/xla/tree/master)?
At least on the surface it looks like they boast PyTorch + TPU.
Would also be cool to know if anyone had experience in running anything with XLA.

Many thanks!


Comments:
Do u mean this one? [link](https://news.developer.nvidia.com/nvidia-achieves-4x-speedup-on-bert-neural-network/)

No, I mean this repo
https://github.com/pytorch/xla/tree/master

Looks like Facebook and Google want to make pytorch on TPU


On May 6, 2019 9:14:25 AM GMT+03:00, chunbo dai <notifications@github.com> wrote:
>Do u mean this one?
>[link](https://news.developer.nvidia.com/nvidia-achieves-4x-speedup-on-bert-neural-network/)
>
>-- 
>You are receiving this because you authored the thread.
>Reply to this email directly or view it on GitHub:
>https://github.com/huggingface/pytorch-pretrained-BERT/issues/583#issuecomment-489510379

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Resetting current_random_doc and current_doc
Author: crowegian
Pull request: no
Labels: name: wontfix

In the class BERTDataset the two variables `self.current_random_doc` and `self.current_doc` are never reset to 0, even when the corpus is closed and reopened. Is it supposed to work this way? I'd think it would run into issues on a small corpus where one counter gets to the same document but the counter is different because it was opened a second time.


https://github.com/huggingface/pytorch-pretrained-BERT/blob/3ae8c8be1e3fc770968cd3fdb3b643e0b166e540/examples/lm_finetuning/simple_lm_finetuning.py#L42-L231

Comments:
Hmm maybe @Rocketknight1 have an insight on this?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
fixing bugs in load_rocstories_dataset in run_openai_gpt.py
Author: sajidrahman
Pull request: yes

The csv reader requires a delimiter argument to read .tsv file in the given example dataset. I've also added link for the dataset and provided a sample eval results in comments. Also, the eval dataset needs to be different from the training dataset, which I've also fixed in the given command to run this script.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/724?src=pr&el=h1) Report
> Merging [#724](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/724?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/commit/98dc30b21e3df6528d0dd17f0910ffea12bc0f33?src=pr&el=desc) will **increase** coverage by `0.05%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/724/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/724?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #724      +/-   ##
==========================================
+ Coverage   62.22%   62.27%   +0.05%     
==========================================
  Files          18       18              
  Lines        3979     3979              
==========================================
+ Hits         2476     2478       +2     
+ Misses       1503     1501       -2
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/724?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_pretrained\_bert/tokenization\_gpt2.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/724/diff?src=pr&el=tree#diff-cHl0b3JjaF9wcmV0cmFpbmVkX2JlcnQvdG9rZW5pemF0aW9uX2dwdDIucHk=) | `83.51% <0%> (+1.06%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/724?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/724?src=pr&el=footer). Last update [98dc30b...63da86c](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/724?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Hi, what is the `run_openai_gpt2_custom.py` file for?

Hi Thomas, I customized the run_openai_gpt.py file to add support for gpt-2. That's why the name might be a bit confusing (run_openai_gpt2_custom.py). I'll better create a separate PR for this file, with a better name.Any suggestion?
Can't save converted checkpoint
Author: Gal1eo
Pull request: no
Labels: name: wontfix

Thank you for creating the pytorch version of BERT. But there is a problem when I use the convert_tf_checkpoint_to_pytorch script, I can't find any files created under the pytorch_dumpy_path.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
run_squad.py without GPU.. Without CUPY
Author: ghost
Pull request: no

I am trying to run_squad.py for QnA (Squad) case. Its dependency is on GPU.. i.e., cupy is to be installed.
In one of my environment I dont have GPU therefore cupy is not getting installed and I am not able to proceed with training.

Can I train on CPU itself?
following is I am trying to run:

```
python run_squad.py \
  --bert_model bert-base-uncased \
  --do_train \
  --do_predict \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad/
```
and I get following error:

```
RuntimeError: CUDA environment is not correctly set up
(see https://github.com/chainer/chainer#installation).No module named 'cupy'
```

Comments:
@SandeepBhutani What was the conclusion of this issue? 

Is this issue still open.. It can be closed.. It was an environment issue..

On Sat, 13 Jul, 2019, 5:00 AM Peter, <notifications@github.com> wrote:

> @SandeepBhutani <https://github.com/SandeepBhutani> What was the
> conclusion of this issue?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/pytorch-pretrained-BERT/issues/137?email_source=notifications&email_token=AHRBKIGTRSIMUPOHIXZNKWDP7EHYNA5CNFSM4GLRLKFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ3DFRY#issuecomment-511062727>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AHRBKIEAQAEZLMPO2JPH64TP7EHYNANCNFSM4GLRLKFA>
> .
>
What is Synthetic Self-Training?
Author: hsm207
Pull request: no
Labels: name: Discussion

name: wontfix

The current best performing model on[ SQuAD 2.0](https://rajpurkar.github.io/SQuAD-explorer/) is BERT + N-Gram Masking + Synthetic Self-Training (ensemble):

![image](https://user-images.githubusercontent.com/2398765/54234467-24466380-454a-11e9-8674-d9e7004da027.png)

What is Synthetic Self-Training?


Comments:
Check Jacob Devlin slides starting from slide 26 [here](https://nlp.stanford.edu/seminar/details/jdevlin.pdf?fbclid=IwAR2TBFCJOeZ9cGhxB-z5cJJ17vHN4W25oWsjI8NqJoTEmlYIYEKG7oh4tlY)

@thomwolf thanks, the slides were helpful. Do you know if there is a recording of the talk publicly available somewhere?

I don't know!

It was a very crowded room and these talks are generally not recorded, sorry…

Is the synthetic self training module in this code?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Same question, will the synthetic self training module be in this code?

Presently not in the library and there are no short-term plans to add synthetic self-training to the library.
[bug] from_pretrained  error with from_tf
Author: shibing624
Pull request: no

https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/modeling.py#L721 , the weights_path should be archive_file, and set from_tf to str is better to load finetuned model, like model name is model.ckpt-25000.meta. 

Comments:
Yeah this is solved in the coming release
Should close the SummaryWriter after using it
Author: t-yaxli
Pull request: no

Really appreciate the good work to implement this package! 

I have tried to run the script: [run_glue.py](https://github.com/huggingface/pytorch-pretrained-BERT/blob/xlnet/examples/run_glue.py). When I test with this script, I found some of the scalars add into the SummaryWriter did not appears in TensorBoard. I think the cause of it is that the code leaves the SummaryWriter unclosed. 



Comments:
Oh yes you are right, thanks it's fixed in the coming release.
XLNet text generation ability
Author: astariul
Pull request: no

Really appreciate the good work to implement XLNet !

I tried running the [XLNet text generation example](https://github.com/huggingface/pytorch-pretrained-BERT/blob/xlnet/examples/generation_xlnet.py)

But the generated text quality is really low. 

Tricks used by https://github.com/rusiaaman/XLnet-gen needs to be added to the example to generate a good example.

---

... Or is it because the pytorch version of XLNet is not fully working yet ?

Comments:
Indeed, I've now added the text padding trick of Aman (add some padding text to have longer inputs) and the quality is really a lot higher.

Will merge the xlnet branch in master and release on Monday.
Performance dramatically drops down without training. 
Author: DariaD
Pull request: no

I use run_classivier and run_squad as it is shown in README. 
If I remove `--do_train` (I already tuned the model and just want to evaluate one more time or with different development set) I expect that result would be the same but performance drops down. 

For example, SQuAD:
with training:  `{"exact_match": 81.35288552507096, "f1": 88.49520505241821}`
without training: `{"exact_match": 0.21759697256385999, "f1": 7.391520686954715}`

I tried my own processor with binary classification (not up to date code though) and without training the value` 1 `only was predicted.

Thank you in advance for any comments.

Comments:
If you want to evaluate only, you have to set `--output_dir` to the path of your previously trained model. Otherwise, the script will use the original model.
XLNet tensor at wrong device issuse
Author: boy2000-007man
Pull request: no

```bash
File "env.xlnet/lib/python3.6/site-packages/pytorch_transformers/modeling_xlnet.py", line 397, in rel_shift
    x = torch.index_select(x, 1, torch.arange(klen))
RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'
```

I meet this issue when using `pytorch-transformers==0.7.0` with multi-gpus, is quickfixed by `x = torch.index_select(x, 1, torch.arange(klen).to(x.device))`

Comments:
This model was WIP. Fixed now.
BERT pretraining routine
Author: yhalk
Pull request: no

Hi,

I was wondering whether the scripts for finetuning can be used to pretrain BERT from scratch on a small dataset that does not require TPUs - is there any difference with the TF pretrain code (different batch sampling or train loss evaluation) other than the TPU support?

Thank you very much in advance!



Comments:
I would not advise to use them for training from scratch. See #751 for discussion and links.
How to fine tune Transformer-XL on own dataset?
Author: Archelunch
Pull request: no
Labels: name: Discussion

name: wontfix

I have my own dataset. What format of data I need to fine tune TransformerXL for text generation?

Comments:
Hi,
You can tokenize your data with the Transformer-XL tokenizer and use it to train your model.

I understand it. But I want to fine tune pretrained model

What is the difference?

Can you start by trying to do what you would do normally to fine-tune a pytorch model and then if you encounter an issue give the error messages back here so we can help?

I found example in readme
And my question - can I fine tune model using this script with my own data? 
` python run_openai_gpt.py \
  --model_name openai-gpt \
  --do_train \
  --do_eval \
  --train_dataset $ROC_STORIES_DIR/cloze_test_val__spring2016\ -\ cloze_test_ALL_val.csv \
  --eval_dataset $ROC_STORIES_DIR/cloze_test_test__spring2016\ -\ cloze_test_ALL_test.csv \
  --output_dir ../log \
  --train_batch_size 16 \ `

Example for gpt, but I guess it is the same for TransformerXL

Hi, you will likely need to adapt this example since Transformer-XL uses memory cells but there is no ready to use example for fine-tuning Transformer-XL in the repo unfortunately (and I don't plan to add one in the near future).

If you want to give it a try feel free to ask more specific questions here.

I would advise to start by reading carefully the author's paper to be sure you have a good understanding of the model. You can also probably start from the authors' own PyTorch training script (which I simplified to make the evaluation script in the present repo).

Is it possible to use the transformer with non-tokenized data @thomwolf? I know the Transformer is built for language modeling, but I would like to take advantage of the self-attention aspect of the Transformer's self-attention mechanism to model continuous data, and wonder how doable that sounds to you? 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to train our own domain-specific data instead of using pre-training models?
Author: yiranxijie
Pull request: no
Labels: name: wontfix

How to train our own domain-specific data instead of using pre-training models?

Comments:
I also have this question whenever someone gets to it, but I think that this isn't doable with this package. There's got to be a way to hack it, but you'd probably have to take away some of the code at the beginning of the pipeline. @yiranxijie 

Is there any news on this? Training one of these models from scratch?

@mattivi not yet

Hi all, so training from scratch will probably never be a goal for the present repo but here are great transformer codebases that were scaled to >64 GPUs:
- XLM: https://github.com/facebookresearch/xlm
- Megatron-LM: https://github.com/NVIDIA/Megatron-LM
- fairseq: https://github.com/pytorch/fairseq

Note that the typical compute required to train BERT is about 64 GPU for 4 days (which currently means around $10k-15k if you are renting cloud compute). TPU training is not possible in PyTorch currently, you should use a TensorFlow repo to do TPU training (like the original BERT or tensor2tensor for instance).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Examples does not work with apex optimizers
Author: tc-yu
Pull request: no

Under fp16 option, optimizer is replaced by one from apex, which does not have the attribute ```get_lr()``` .

https://github.com/huggingface/pytorch-pretrained-BERT/blob/98dc30b21e3df6528d0dd17f0910ffea12bc0f33/examples/run_squad.py#L315-L317

Should be able to reproduce the error by running the example [here](https://github.com/huggingface/pytorch-pretrained-BERT#fine-tuning-bert-large-on-gpus) in the README, error message should be something like ```AttributeError, FP16_Optimizer does not have attribute get_lr()```

Comments:
You can  use #  on line 316 and it can solved this problem.
Fine tuning time did not change much after freezing layers 
Author: kbulutozler
Pull request: no
Labels: name: wontfix

Hi, 

I am using simple_lm_finetuning.py to fine tune the model. I wanted to freeze all parameters from the very beginning to the beginning of the 12th transformer layer, I looked into parameters by name and used a counter, took the value of the counter corresponding to the beginning of the 12th layer, and used this value to freeze all layers before 12th layer. You can understand better from this piece of code I added to simple_lm_finetuning.py. 

```
ctr = 0 
for name, param in model.named_parameters():
    ctr += 1
    #print(ctr)
    #print(name)
    if(ctr < 183): #183 is where 12th transformer layer starts
        param.requires_grad = False
```

With Nvidia K80 and original simple_lm_finetuning.py, one epoch required about 37-38 hours. After adding this piece of code, it required about 28 hours. Since I have frozen all parameters from the beginning to the 12th layer, I was expecting more reduced time. Where am I wrong?

I am open to other suggestions of fine tuning methods that requires less computational time. 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
attention_mask at run_squad.py
Author: seanie12
Pull request: no

I think there's minor mistake in [run_squad.py](https://github.com/huggingface/pytorch-transformers/blob/5fe0b378d8/examples/run_squad.py#L298) at line 298
```
 inputs = {'input_ids': batch[0], 
'token_type_ids':  None if args.model_type == 'xlm' else batch[1],  
'attention_mask':  batch[2],
'start_positions': batch[3],
'end_positions':   batch[4]}
```
but i think batch[1] is attention_mask and batch[2] is segment_ids, so it should be like this
``` 
inputs = {'input_ids': batch[0],
'attention_mask':  batch[1], 
'token_type_ids':  None if args.model_type == 'xlm' else batch[2],  
'start_positions': batch[3], 
'end_positions':   batch[4]}
```
because the data is 
```
dataset = TensorDataset(all_input_ids, all_input_mask, all_segment_ids,
                                all_start_positions, all_end_positions,
                                all_cls_index, all_p_mask)

```


Comments:
I have same issue. 

Thanks @seanie12!
AssertionError in BERT-Quickstart example
Author: marcalt94
Pull request: no

Hey

I tried running the Quickstart example with my own little text. Everything works fine until I get to the ```assert tokenized_text ==... ``` part. When I try to enter my text instead of the Jim Henson text, I get the following error message: ```Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError ```

I'm not sure if the error is an operation-problem or if it is an issue...

Comments:
What is the full line of the assertion you are testing?
And what is your input text?

I think the mistake was with me, sorry