noamaanMulla-03/transformers-issues huggingface dataset:
Using BERT with custom QA dataset
Author: gqoew
Pull request: no

Hi,

I want to use BERT to train a QA model on a custom SQuAD-like dataset. Ideally, I would like to leverage the learning from the SQuAD dataset, and add fine-tuning on my custom dataset, which has specific vocabulary.

What is the best way to do this?

Comments:
I think that you should start by pretraining a BERT model on SQuAD to give it a sense on how to perform question answering and then try finetuning it to your task. This may already give you good results, if it doesn't you might have to dig a bit deeper in the model.

I don't really know how adding your domain specific tokens to the vocabulary would interact with the tokenizer.

One nice recent example is "A BERT Baseline for the Natural Questions" by Chris Alberti, Kenton Lee and Michael Collins from Google Research: http://arxiv.org/abs/1901.08634

This might help other dev who want to use BERT for custom QA: https://github.com/cdqa-suite/cdQA
Any way to reduce the model size to <250mb?
Author: bolaft
Pull request: no
Labels: name: wontfix

Google Cloud's online prediction service has a 250mb limit for uploaded models. I don't think I have ever seen a BERT model that small. Casting all tensors to half precision reduces the model to ~350mb, is there any way to go even further than that?

Comments:
Probably not - it would certainly be possible to make a smaller BERT model that would fit into this size, but all of the available pre-trained models have too many parameters, so you'd have to train it from scratch (which is very slow, and isn't something this repo supports yet).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
a single sentence classification task, should the max length of sentence limited to half of 512, that is to say 256
Author: alphanlp
Pull request: no
Labels: name: Discussion

name: wontfix

hi, if i have a single sentence classification task, should the max length of sentence limited to half of 512, that is to say 256?

Comments:
Why should it be limited to half of 512?

> Why should it be limited to half of 512?

cause when do train,  we have sentence embedding 0 and 1, but in a single sentence classification task ,we just embedding 0, if this get bad influence

You can just set the whole sequence to sentence 0. Create a DataProcessor class for your task and set the whole input sequence to `text_a`, example:

```
class MyProcessor(DataProcessor):
# some other methods here

    def _create_examples(self, lines, set_type):
        """Creates examples for the training and dev sets."""
        examples = []
        for (i, line) in enumerate(lines):
            guid = "%s-%s" % (set_type, i)
            text_a = line[1]
            label = line[0]
            examples.append(
                InputExample(guid=guid, text_a=text_a, text_b=None, label=label))
        return examples
``` 
Notice `text_b=None`.

How should I do if I have not only a sentence, but a whole text? 
I don't clearly understand how to extend `BertForSequenceClassification` with my own dataset for training/evaluating.
I have a dataset consisting of text/label pairs, where text can have multiple sentences. 

Just send in the whole text as one "sentence", the limit on a sequence length that can be sent at once to BERT is 512 tokens

Ok, thanks.
One more question related to classification. BERT tokenizes my sentences pretty strange:
> 04/27/2019 16:08:32 - INFO - __main__ -   tokens: [CLS] @ bra ##yy ##yy ##ant Так акт ##иви ##ровала ##сь новая карта , ст ##ара ##я и была не ##ак ##тив ##на . [SEP]

Why are more than a half of the words a separated with `#`? I mean, these words are on russian and many of them are splited to several parts with `#`, though it is one word. Should this be fixed during training?

That's the WordPiece tokenization, its a way to match subwords when an out of vocabulary word is encountered. It's explained in the bert paper with references. It's as it should be. 

Ok, thank you so much.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Pad inputs to multiple of 8
Author: thomwolf
Pull request: no
Labels: name: wontfix

Pad transformer's inputs to multiple of 8 to better use Tensorcores in fp16 mode.

@glample's [XLM](https://github.com/facebookresearch/XLM) does that and it seems still relevant with CUDA 10 (cc @yaroslavvb).

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
should loss_scale be multiplied to the loss explicitly?
Author: Jim-Song
Pull request: no
Labels: name: wontfix

I noticed that in the run_swag.py, the following code is included

    if args.fp16 and args.loss_scale != 1.0:
        # rescale loss for fp16 training
        # see https://docs.nvidia.com/deeplearning/sdk/mixed-precision-training/index.html
        loss = loss * args.loss_scale

and in run_squad.py,  this is not included.
the optimizer in them are identity, so which is right?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to get back input and predictions as string
Author: bikashg
Pull request: no
Labels: name: wontfix

Once I am done fine tuning my `BertForSequenceClassification` model, I evaluate it on a validation set. I can see the loss and accuracy scores but I would also like to get the actual labels (as string) it predicted for each sentence (string) in the validation dataset. How could I do that? 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Convert_tf_checkpoint_to_pytorch for bert-joint-baseline
Author: raheja
Pull request: no
Labels: name: Help wanted

name: Discussion

name: wontfix

Hello,

I know that the format of Squad and Google NQ is different, but is there a way to convert the bert joint model for Natural Questions (https://github.com/google-research/language/tree/master/language/question_answering/bert_joint) to pytorch? I get this error
'BertForPreTraining' object has no attribute 'answer_type_output_bias'

Comments:
I also encountered a similar problem AttributeError: 'BertForPreTraining' object has no attribute 'crf_loss'

@thomwolf   

Looking forward to your reply

Hi, from my reading of the [Natural Questions model](https://github.com/google-research/language/blob/master/language/question_answering/bert_joint/run_nq.py#L879-L932) it doesn't seems to be directly possible to load this model in the current library (with simple hacks).

You will need to define a new sub-class of `BertModel` (e.g. `BertForNaturalQA`) that reproduce the architecture of the TensorFlow model I pointed to. If you use the same name as the TensorFlow variables for the attributes of your PyTorch model you should be able to load the model with the current loading script.

I don't have time to do this right now but if you want to start opening a PR, I can review it.

Basically, just add another class after the `BertForQuestionAnswering` class in [`modeling.py`](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/modeling.py#L1130)

Thanks a lot for the reply. Will try this out.

Best,


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hi @raheja, do you have any updates related to this? I'm trying to do the same and would welcome any hint!  

Hi @paulachocron 
Haven't been able to try or implement this yet. I am still using the tf version for now.
TypeError: '<' not supported between instances of 'NoneType' and 'int'
Author: quocnle
Pull request: no
Labels: name: wontfix

Hi, I am trying to do classification fine tuning using bert-base-uncased.  I am using examples from master and pytorch_pretrained_bert==0.6.2.  Here are my repro steps:

1.  I create a train.tsv and dev.tsv file with my own domain data.  The files contain sentences and labels separated by a tab.  I put these files in /tmp/bertdata

2.  I do fine tuning using: python run_classifier.py --data_dir /tmp/bertdata --bert_model bert-base-uncased --task_name sst-2 --do_lower_case --do_train --output_dir tmp.  This works fine and a model, config json, and vocab.txt are placed in tmp

3. I try to use the fine tuned model on the dev.tsv set:  python run_classifier.py --data_dir /tmp/bertdata --bert_model tmp --task_name sst-2 --do_lower_case --do_eval --output_dir tmp_result.  When I do that, I get this error:

Traceback (most recent call last):
  File "run_classifier.py", line 1024, in <module>
    main()
  File "run_classifier.py", line 794, in main
    t_total=num_train_optimization_steps)
  File "/home/ec2-user/anaconda3/lib/python3.6/site-packages/pytorch_pretrained_bert/optimization.py", line 215, in __init__
    schedule = schedule_type(warmup=warmup, t_total=t_total)
  File "/home/ec2-user/anaconda3/lib/python3.6/site-packages/pytorch_pretrained_bert/optimization.py", line 45, in __init__
    if t_total < 0:
TypeError: '<' not supported between instances of 'NoneType' and 'int'

Anything obvious I am doing wrong?  Thanks!

Comments:
I also get this problem when predicting，Did you  solved the problem？

Here is the problem during initialization of the optimizer:
` t_total=num_train_optimization_steps)`

This var is initialized with `None` for the first time `num_train_optimization_steps = None`
and it's initialized correctly only when `--do_train` flag is passed to the script
```
    if args.do_train:
        train_examples = processor.get_train_examples(args.data_dir)
        num_train_optimization_steps = int(
            len(train_examples) / args.train_batch_size / args.gradient_accumulation_steps) * args.num_train_epochs
        if args.local_rank != -1:
            num_train_optimization_steps = num_train_optimization_steps // torch.distributed.get_world_size()
```
but in case of `--do_eval` this var is `None` and you got an error from description.

It's a bug, I think, and should be fixed. For you local needs, just initialize the optimizer "somehow" - it's not used while evaluating.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
fix evaluation bug
Author: leimao
Pull request: yes

The original `run_squad.py` has a potential bug. If we only want to run the script to do evaluation, the model will not be properly loaded. The simple fix is provided.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/745?src=pr&el=h1) Report
> Merging [#745](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/745?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/commit/dad3c7a485b7ffc6fd2766f349e6ee845ecc2eee?src=pr&el=desc) will **decrease** coverage by `0.05%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/745/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/745?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #745      +/-   ##
==========================================
- Coverage   62.27%   62.22%   -0.06%     
==========================================
  Files          18       18              
  Lines        3979     3979              
==========================================
- Hits         2478     2476       -2     
- Misses       1501     1503       +2
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/745?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_pretrained\_bert/tokenization\_gpt2.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/745/diff?src=pr&el=tree#diff-cHl0b3JjaF9wcmV0cmFpbmVkX2JlcnQvdG9rZW5pemF0aW9uX2dwdDIucHk=) | `82.44% <0%> (-1.07%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/745?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/745?src=pr&el=footer). Last update [dad3c7a...64b2a82](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/745?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Update run_squad.py
Author: saikalyan9981
Pull request: yes

model = BertForQuestionAnswering.from_pretrained(args.bert_model) is written twice.
I think the else part is redundant there

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/707?src=pr&el=h1) Report
> Merging [#707](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/707?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/commit/c304593d8fa93f25febe1458c63497a846749c89?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/707/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/707?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master     #707   +/-   ##
=======================================
  Coverage   62.27%   62.27%           
=======================================
  Files          18       18           
  Lines        3979     3979           
=======================================
  Hits         2478     2478           
  Misses       1501     1501
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/707?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/707?src=pr&el=footer). Last update [c304593...620f2c1](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/707?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Yes, that's for clarity
Added option to use multiple workers to create training data
Author: ceremonious
Pull request: yes

Added a command line argument to allow using a multiprocessing pool to generate training data for all the epochs at once. 

The shelve object isn't pickleable, so it can't be used with the Pool

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/733?src=pr&el=h1) Report
> Merging [#733](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/733?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/commit/98dc30b21e3df6528d0dd17f0910ffea12bc0f33?src=pr&el=desc) will **increase** coverage by `0.05%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/733/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/733?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #733      +/-   ##
==========================================
+ Coverage   62.22%   62.27%   +0.05%     
==========================================
  Files          18       18              
  Lines        3979     3979              
==========================================
+ Hits         2476     2478       +2     
+ Misses       1503     1501       -2
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/733?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_pretrained\_bert/tokenization\_gpt2.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/733/diff?src=pr&el=tree#diff-cHl0b3JjaF9wcmV0cmFpbmVkX2JlcnQvdG9rZW5pemF0aW9uX2dwdDIucHk=) | `83.51% <0%> (+1.06%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/733?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/733?src=pr&el=footer). Last update [98dc30b...08ff056](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/733?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Nice, thanks!
Transformer XL from Pytorch model
Author: agemagician
Pull request: no
Labels: name: wontfix

Hello,

I have trained the original pytorch version of transformer xl, and I want to load it to get the hidden state and prediction.

However, it doesn't work. Apparently you only support to load a model from TensorFlow model checkpoints only.

Is there any hint or feature modification to make it work with model.pt and cache.pt ?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
the size of words and the size of lables do not match
Author: g-jing
Pull request: no
Labels: name: wontfix

When I run bert-large-cased model, it prints "the size of words and the size of lables do not match" but get no error message. What is this issue? Thanks

Comments:
Can you give the exact log of (and before) the error message?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
performance does not change but loss decrease
Author: g-jing
Pull request: no
Labels: name: wontfix

After training bert-lstm-crf model for 25 epoches, the performance on training set
 here is the performance on train set, dev set and test set:

25th epoch: 
tensor(10267.6279, device='cuda:0')
(0.42706720346856614, 0.4595134955014995, 0.4426966292134832)
(0.43147208121827413, 0.4271356783919598, 0.42929292929292934)
(0.4460093896713615, 0.4668304668304668, 0.4561824729891957)

26th epoch:
tensor(10219.3398, device='cuda:0')
(0.44544364508393286, 0.4951682772409197, 0.46899163642101943)
(0.4469135802469136, 0.4547738693467337, 0.45080946450809467)
(0.45871559633027525, 0.4914004914004914, 0.4744958481613286)

27 epoch:
tensor(10169.0742, device='cuda:0')
(0.44544364508393286, 0.4951682772409197, 0.46899163642101943)
(0.4469135802469136, 0.4547738693467337, 0.45080946450809467)
(0.45871559633027525, 0.4914004914004914, 0.4744958481613286)

more epochs:
......(same performance but lower loss)

And here is the main code:
for epoch in tqdm(range(200)):
    loss = train_one_epoch(dataloader=source_train_dataloader,
                           model=model, optimizer=optimizer)
    train_perf = test_one_epoch(dataloader=source_train_dataloader_for_test,
                                model=model)
    dev_perf = test_one_epoch(dataloader=source_dev_dataloader, model=model)

    test_perf = test_one_epoch(dataloader=source_test_dataloader, model=model)
    base_result_loc = "bert_char_ps/bert_char_result"
    # store performance result
    add_model_result(
        base_result_loc,
        epoch,
        loss,
        train_perf,
        dev_perf,
        test_perf)

I can't not figure out why the loss keep decreasing but the performance on train set, dev set and test set keep unchanged. I have been stuck in this for a few days. Anyone knows how to handle this?  It would be of great help.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Results of Fine-tuned model changes in every run
Author: cagrikymk
Pull request: no
Labels: name: wontfix

After I load the model with:
`
model = BertForNextSentencePrediction.from_pretrained("bert-base-uncased",state_dict=model_state_dict)
model.eval()
`
The prediction results are not stable. They change dractically in every run.

It gets stable if I fix the seed but I dont understand why we need that. Isnt the model supposed to be fixed since we are just evaluating?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
about pytorch 1.1.0 rerlease
Author: yeontaek
Pull request: no
Labels: name: wontfix

Hi today pytorch 1.1.0 release(https://github.com/pytorch/pytorch/releases/tag/v1.1.0)

In version 1.1.0, added a new module implementing Multi-headed-Attention.

And various bugs have been modified.

Do you plan to update to suit that version?




Comments:
Hi,

The repo is compatible with PyTorch 1.1.0.

But, we probably won't switch to PyTorch Multi-headed-Attention module since this would mean refactoring all the models and adding complexity to the tensorflow conversion codes for unclear gains.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How many datasets does Bert use in pretraining process?
Author: DecstionBack
Pull request: no
Labels: name: Discussion

name: wontfix

Hi all,
I try to generate the pretraining corpus for BERT with pregenerate_training_data.py. In the BERT paper, it reports about 6M+ instances(segment A+segmentB, less than 512 tokens). But I get 18M instances, which is almost 3 time than BERT uses. Does anyone have any idea on the result and does anyone know if I need to process WikiPedia and BookCorpus first and then try to generate training instances?  Thanks very much in advance!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
BERT pre-training using only domain specific text
Author: nightowlcity
Pull request: no
Labels: name: wontfix

BERT is pre-trained using Wikipedia and other sources of normal text, but my problem domain has a very specific vocabulary & grammar. Is there an easy way to train BERT completely from domain specific data (preferably using Keras)?
 
The amount of pre-training data is not issue and we are not looking for the SOTA results. We would do fine with a smaller scale model, but it has to be trained from our data.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.