noamaanMulla-03/transformers-issues huggingface dataset:
Potential redundancy in run_classifier.py example script
Author: ZhaofengWu
Pull request: no
Labels: name: Discussion

name: wontfix

https://github.com/huggingface/pytorch-pretrained-BERT/blob/7cc35c31040d8bdfcadc274c087d6a73c2036210/examples/run_classifier.py#L641-L642

Here we are calling the model twice. I understand that the model returns different things depending on the presence of `label_ids`, but this could actually be quite expensive. I think we can change the `if label is not None` branch in the model code below to return two items instead, but I'm not sure if it will break things elsewhere.

https://github.com/huggingface/pytorch-pretrained-BERT/blob/7cc35c31040d8bdfcadc274c087d6a73c2036210/pytorch_pretrained_bert/modeling.py#L969-L979


Comments:
I agree with you, this part of the API could be improved.

The BERT model is now used in several third-party libraries like AllenNLP and FLAIR so we have to be careful not to make any breaking change on this model.

We could add a flag to get full output maybe.

Mind if I submit a PR adding a flag that allows getting the full output in the case of `if labels is not None`?

I'm ok to welcome a PR on that but there is one issue with flags in `forward()` call and multi-GPU you may or may not be aware of and we need to think about:
All the inputs to the `forward()` call are split across GPUs so having a non-batched input like a flag break DataParallel for multi-gpu.
So maybe we need to add a general flag in the models which can be set. Maybe you can try to draft a PR and we check it behave well on the examples then.
Maybe you can also have a deep look at the way inputs are split in DataParallel and check whether a  solution with flag in the arguments of the `forward()` call work.

Hmm, okay, I was not aware of that. I think adding it as a property on the model itself is arguably an unnecessary over-complication for a simple matter. Returning a tuple value is much cleaner and the user can always discard the information that they don't care. Although I do understand the concern of backward compatibility. So I won't do this PR and will let you guys make the decision between the two options. Personally I lean more towards the API-breaking option.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Can the use of [SEP] reduce the information extraction between the sentences?
Author: RomanShen
Pull request: no

Hello. I know that [CLS] means the start of a sentence and [SEP] makes BERT know the second sentence has begun. [SEP] can’t stop one sentence from extracting information from another sentence. However, I have a question.
If I have 2 sentences, which are s1 and s2, and our fine-tuning task is the same. In one way, I add special tokens and the input looks like [CLS]+s1+[SEP] + s2 + [SEP]. In another, I make the input look like [CLS] + s1 + s2 + [SEP]. When I input them to BERT respectively, what is the difference between them? Will the s1 in second one integrate more information from s2 than the s1 in first one does? Will the token embeddings change a lot between the 2 methods?
Thanks for any help!

Comments:
I think so. Ultimately you should have s1 and s2 in input, your [CLS] + s1 + s2 + [SEP] will be equivalent to `[CLS] + s1 + [SEP]` in `[CLS] + s1 + [SEP] + s2 + [SEP]` where `s1` now is the concatenation of `s1` and `s2`. I don't think that's what you want to do.
What is the use of [SEP]?
Author: RomanShen
Pull request: no

Hello. I know that [CLS] means the start of a sentence and [SEP] makes BERT know the second sentence has begun. [SEP] can’t stop one sentence from extracting information from another sentence. However, I have a question.
If I have 2 sentences, which are s1 and s2., and our fine-tuning task is the same. In one way, I add special tokens and the input looks like [CLS]+s1+[SEP] + s2 + [SEP]. In another, I make the input look like [CLS] + s1 + s2 + [SEP]. When I input them to BERT respectively, what is the difference between them? Will the s1 in second one integrate more information from s2 than the s1 in first one does? Will the token embeddings change a lot between the 2 methods?
Thanks for any help!

Comments:
@RomanShen What is your observation on your question
run_squad.py cannot predict only
Author: lixinsu
Pull request: no
Labels: name: Help wanted

name: wontfix

Existing code cannot load fine-tuned model properly.
https://github.com/huggingface/pytorch-pretrained-BERT/blob/f3e5404880902a1bdfed2b1d47d10a6c672dc430/examples/run_squad.py#L1011-L1025

Comments:
It is better to imitate original Bert repo to add separate argument `args.vocab_file`, and during prediction,  argument `bert_model` is the directory containing the fine-tuned model.   

Make sense indeed. Would you like to submit a PR on that?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Incrementally Train BERT with minimum QnA records - to get improved results
Author: ghost
Pull request: no
Labels: name: Discussion

name: wontfix

This question is posted in stackexchange too, but is pointing to BERT group:
https://datascience.stackexchange.com/questions/47406/incrementally-train-bert-with-minimum-qna-records    
Question is after training on my data on some new questions and answers, new checkpoints are generated. With new checkpoints, when asked same question the answer is not correct, and is wrong again. Why is training not helping making answer right?  
  
Though the questions are point to tensorflow version, but same is tried in pytorch version too and results are same. Some experts on BERT or transformers or neural network probably can better point the issue. 

Details:
We are using Google BERT for Question and Answering. We have fine tuned BERT with SQUAD QnA release train data set (https://github.com/google-research/bert , https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json)

It generated new checkpoints and BERT is giving good answers for most of questions we asked on our text documents. However, there are some questions which it is answering wrong, so we are trying to further fine tune with our Question and known answer on our text document. We further trained based on last generated checkpoint and got new checkpoint.

With new checkpoint when we are asking the same question, the answer did not got corrected! Previously BERT was giving wrong answer with 99% confidence and now also giving same wrong answer with 95% confidence.

Can someone suggest, if they have same or similar experience, and suggest please.
Following are questions in BERT github Issues, and are unanswered for quite some time:

BERT accuracy reduced after providing custom training..The answer is also not correct : https://github.com/google-research/bert/issues/492
Unable to incrementally train BERT with custom training: https://github.com/google-research/bert/issues/482
Little training has no impact: https://github.com/google-research/bert/issues/481
Custom Domain Training: https://github.com/google-research/bert/issues/498

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Wondering Noone is replying

On Sat, 25 May, 2019, 4:09 PM stale[bot], <notifications@github.com> wrote:

> Closed #384
> <https://github.com/huggingface/pytorch-pretrained-BERT/issues/384>.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/pytorch-pretrained-BERT/issues/384?email_source=notifications&email_token=AHRBKICW2VUPHXBBGSQD5O3PXEJN5A5CNFSM4G666UB2YY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGORUM5MZY#event-2367280743>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AHRBKIG2BUNIOBDUV6QGHQDPXEJN5ANCNFSM4G666UBQ>
> .
>
GPT-2 Tokenizer error!
Author: PhungVanDuy
Pull request: no

I tried to use GPT-2 to encode with `text = "This story gets more ridiculous by the hour! And, I love that people are sending these guys dildos in the mail now. But… if they really think there's a happy ending in this for any of them, I think they're even more deluded than all of the jokes about them assume."` but it error. 

![Screenshot from 2019-05-24 14-31-30](https://user-images.githubusercontent.com/28798474/58310361-a8fa0b80-7e30-11e9-9554-8d34c2b9f968.png)


Comments:
Maybe you can find the solution at #537 

> Maybe you can find the solution at #537

Thank for your support! It works for me!
No softmax activation in BertForTokenClassification
Author: shudima
Pull request: no

The BertForTokenClassification class has the `classifier` member, which is a linear layer. 
In the `forward` function, it treats it's out as probabilities (Cross Entropy for loss) but there's no softmax. Is there a reason for that? 

Comments:
It's because `nn.CrossEntropyLoss` already has a Softmax integrated in the module:
https://pytorch.org/docs/stable/nn.html?highlight=crossentropy#torch.nn.CrossEntropyLoss

I see it now. Thanks!
The prediction accuracy for the masked token is ZERO when using the pretrained model. Does it make sense?
Author: sszzsupersupersupersuper
Pull request: no

Hi, this question might be not relavent to the code but to BERT, I still hope I can find someone to help me out. 

I run the pretrained model BertForPreTraining and test it on my own text data. Because BERT has knowledge about language so I expect it to be able to predict the masked tokens with a reasonable accuracy, however it is zero. I run the model with model.eval() mode. Does it make sense that it predicts the masked tokens with zero? 

Thanks for the help.

Comments:
max_seq_length for squad
Author: ZhaofengWu
Pull request: no

The example script for squad sets `--max_seq_length` at 384 as default. However it seems that many paragraphs in squad exceed this length. Then what if the answer to some question lies in the truncated part of the paragraph?

Comments:
Then you're doomed for this answer. There are a possibility to do a sliding window approach but we didn't implemented it in the examples of pytorch-pretrained-bert.
Check this issue (and the linked TensorFlow issue) for a discussion on this: https://github.com/huggingface/pytorch-pretrained-BERT/issues/89
AttributeError: 'BertOnlyMLMHead' object has no attribute 'seq_relationship'
Author: leejason
Pull request: no
Labels: name: Need more information

name: wontfix

Is there way to fix it? 
```bash
Skipping cls/predictions/transform/dense/kernel/adam_m
Skipping cls/predictions/transform/dense/kernel/adam_v
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-11-5d7ca59bd98d> in <module>()
      7 model = BertForMaskedLM.from_pretrained(
      8     pretrained_model_name_or_path=model_folder,
----> 9     from_tf=True, cache_dir=None)
     10 #model = BertForMaskedLM.from_pretrained(model_version)
     11 

/content/my_pytorch-pretrained-BERT/pytorch_pretrained_bert/modeling.py in from_pretrained(cls, pretrained_model_name_or_path, state_dict, cache_dir, from_tf, *inputs, **kwargs)
    605             # Directly load from a TensorFlow checkpoint
    606             weights_path = os.path.join(serialization_dir, TF_WEIGHTS_NAME)
--> 607             return load_tf_weights_in_bert(model, weights_path)
    608         # Load from a PyTorch state_dict
    609         old_keys = []

/content/my_pytorch-pretrained-BERT/pytorch_pretrained_bert/modeling.py in load_tf_weights_in_bert(model, tf_checkpoint_path)
    100                 #
    101             else:
--> 102                 pointer = getattr(pointer, l[0])
    103                 '''
    104                 #J

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __getattr__(self, name)
    533                 return modules[name]
    534         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 535             type(self).__name__, name))
    536 
    537     def __setattr__(self, name, value):

AttributeError: 'BertOnlyMLMHead' object has no attribute 'seq_relationship'
```

Comments:
Hi,
What command are you running to load the model?
Are you loading your own model or one or our pre-trained one?

It's normal that there is no `seq_relationship` attribute in a `BertOnlyMLMHead` but our pre-trained model should load without error.


I was loading my own pre-trained model by "convert_tf_checkpoint_to_pytorch.py". Would it be ok if I skip "seq_relationship" in the source code? 

Maybe, I don't have enough information to really be able to help you at this stage.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
AllenNLP TransformerXL
Author: DataDaveH
Pull request: no
Labels: name: Discussion

name: wontfix

Has anyone done any work on wrapping up TransformerXL for AllenNLP?

Comments:
Maybe you should ask in the AllenNLP repo as well?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
gpt2 tokenizer issue with ValueError: chr() arg not in range(256) in Python 2.X
Author: KaiQiangSong
Pull request: no
Labels: name: Help wanted

name: wontfix

See [the code](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/tokenization_gpt2.py#L50)

Here's a solution for python 2.X
```python
@lru_cache()
def bytes_to_unicode():
    """
    Returns list of utf-8 byte and a corresponding list of unicode strings.
    The reversible bpe codes work on unicode strings.
    This means you need a large # of unicode characters in your vocab if you want to avoid UNKs.
    When you're at something like a 10B token dataset you end up needing around 5K for decent coverage.
    This is a signficant percentage of your normal, say, 32K bpe vocab.
    To avoid that, we want lookup tables between utf-8 bytes and unicode strings.
    And avoids mapping to whitespace/control characters the bpe code barfs on.
    """
    bs = list(range(ord("!"), ord("~")+1))+list(range(ord("¡"), ord("¬")+1))+list(range(ord("®"), ord("ÿ")+1))
    cs = bs[:]
    n = 0
    for b in range(2**8):
        if b not in bs:
            bs.append(b)
            cs.append(2**8+n)
            n += 1
    cs = [unichr(n) for n in cs]
    return dict(zip(bs, cs)
```
And Change the encode to be
```python
    def encode(self, text):
        bpe_tokens = []
        for token in re.findall(self.pat, text):
            token = ''.join(self.byte_encoder[ord(b)] for b in token.encode('utf-8'))
            bpe_tokens.extend(self.encoder[bpe_token] for bpe_token in self.bpe(token).split(' '))
        if len(bpe_tokens) > self.max_len:
            raise ValueError(
                "Token indices sequence length is longer than the specified maximum "
                " sequence length for this OpenAI GPT-2 model ({} > {}). Running this"
                " sequence through the model will result in indexing errors".format(len(bpe_tokens), self.max_len)
            )
        return bpe_tokens
```

Comments:
Indeed, I have added backward compatibility to python 2 for GPT-2.
Do you want to submit a PR on this?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
slow training speed even 20 steps
Author: KavyaGujjala
Pull request: no
Labels: name: wontfix

Hi,
I am running run_lm_finetuning.py code on a 1 million sentences .
It's taking more than 20 hours even for each epoch. 

where as run_pretraining.py code from google-research/bert takes very less time.

What can be the reason?

How to resolve this?

Comments:
We have new lm_fintetuning scripts thanks to @Rocketknight1 PR #392.
Maybe you can try these ones.
They are in the `./examples/lm_finetuning/` folder.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
something wrong in example
Author: Wangpeiyi9979
Pull request: no
Labels: name: Need more information

name: wontfix


![image](https://user-images.githubusercontent.com/42565075/54999939-5789f780-500c-11e9-958f-1c7b0b92a257.png)

the segmentation is wrong。。

Comments:
Do you have the latest `pytorch-pretrained-bert` ?
```python
import pytorch_pretrained_bert
pytorch_pretrained_bert.__version__
```

Thank you so much!!!!!!!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Empty nbest_predictions.json for run_squad.py
Author: luyang-ai4med
Pull request: no

This is due to extra indentation on line 623 in run_squad.py 
It should be outside of the "if else" loop. 

Comments:
Good catch! Do you want to submit a PR? Otherwise, I'll fix it in the next release.

Hi @thomwolf 
The issue still persists, there were two extra indentations and you removed only one to move the line out of inner if-else but, one more indentation should be removed to bring [L620](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_squad.py#L620) out of outer if-else. Doing this produces non-empty `nbest_predictions.json` file.
Multilabel classification and diverging loss
Author: nicolas-mng
Pull request: no

Hi,

I'm not sure I'm posting this at the right spot but I am trying to use your excellent implementation to do some multi label classification on some text. I basically adapted the run_classifier.py code to a Jupyter Notebook and change a little bit the BERT Sequence Classifier model so it can handle multilabel classification. However, my loss tends to diverge and my outputs are either all ones or all zeros.

The labels distribution in my train dataset is :

`array([ 65, 564, 108,  17,  40,  26, 306, 195,  25, 345,  54,  80, 214]) ` 

i.e. the label 1 is used 65 times, the label 2 is used 564 times etc... Each sample has between 1 and 4 labels.

I am using the Adam Optimizer on the BCEWithLogitsLoss and I am unable to figure out where the problem comes from? Should I add some weights in my loss function? Do I use it in a right way? Is my model wrong somewhere? I attach to this post a Notebook of my test. Maybe someone encountered the same problem before and could help me?

[NOTEBOOK](https://nbviewer.jupyter.org/github/nicolas-mingione/nlp/blob/master/test_bert.ipynb)

Thanks !

Comments:
Hey, I am working on something similar. I feel like the original code might be incorrect. They seem to directly take the output of the model as 'loss' without applying any criteria. But I might be totally wrong. 

Hey! :)

Really? What do you mean by criteria?

I tried to artificially change my dataset so that the target outputs are
[1, 0, 0,..., 0] for every sample. I wanted to see whether the model was
able to learn this dummy case. However, it fails and it predicts the exact
opposite, i.e. [0, 1, 1,.... 1]. That's why I think I must be doing
something wrong somewhere.


Hi @nicolas-mng @zhipeng-fan Did you guys manage to get a multilabel problem to work ? Could you please share a gist ?

Hi, you should have a look here: https://medium.com/huggingface/multi-label-text-classification-using-bert-the-mighty-transformer-69714fa3fb3d ;)
Distributed Training Gets Stuck
Author: moinnadeem
Pull request: no
Labels: name: Help wanted

name: Need more information

name: wontfix

Hi,

When I freeze the BERT layers, distributed training works just fine. However, when I unfreeze the BERT layers, the first node continues training, and all other nodes wait on the training step with 100% GPU utilization on the first GPU. Is this expected behavior, or am I doing something wrong?

I'm using PyTorch 1.0.1post2, and 8 GTX 1080 GPUs per machine.

Best,
Moin

Comments:
Hi,
One possible cause of this behavior may be the way you are freezing your parameters.

PyTorch's `DistributedDataParallel` is a rather sensitive beast as you can juge by the number of warnings in [its doc](https://pytorch.org/docs/stable/nn.html#torch.nn.parallel.DistributedDataParallel).

Two features are especially important to keep in mind:
- *You should never try to change your model’s parameters after wrapping up your model with DistributedDataParallel* or unexpected behaviors can happen, since some parameters’ gradient reduction functions might not get called.
- Another that is probably not important for you but you should be award of: Constructor, forward method, and differentiation of the output (backward pass) are *distributed synchronization point*.

So maybe you are freezing your parameters after having wrapped up your model with DistributedDataParallel?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
pytorch model to tensorflow checkpoint
Author: KavyaGujjala
Pull request: no
Labels: name: wontfix

How to convert a pytorch_model.bin to tensorflow checkpoint?

Comments:
Hi, there is no script to do that currently. I don't plan to add this feature in the short term but I would be happy to welcome  a PR on that.

A PR for this would be great. It would allow a simple deployment via Han's bert-as-service
 https://github.com/hanxiao/bert-as-service/

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Unable to incrementally train BERT with custom training
Author: AmrHendy
Pull request: no
Labels: name: Discussion

name: wontfix

I have trained BERT with custom small training. I am unable to train the same on QQP and then with custom train. Any discussion will be appreciated.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
DistributedDataParallel Not Working
Author: moinnadeem
Pull request: no
Labels: name: Need more information

name: wontfix

Hi, I've been stuck on this for days, so I decided to make an issue.

When I run DistributedDataParallel with the PyTorch launch module, I see that one machine will start training without waiting for the other one to start; this is different than if I run it without the launch module. WIthout the launch module, I am also letting one process have access to multiple GPUs, rather than just one.

I'm following your implementation of DistributedDataParallel in your Medium article: https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255

Comments:
Hi @moinnadeem,
What is the hardware you are using and what is the exact command you are using to run DistributedDataParallel with the PyTorch launch module? 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
warmup for BertAdam
Author: yangdechuan
Pull request: no

https://github.com/huggingface/pytorch-pretrained-BERT/blob/ee0308f79ded65dac82c53dfb03e9ff7f06aeee4/examples/run_classifier.py#L860

BertAdam() can update learning rate by itself.
 Why update learning rate manually here?

Comments:
Because we don't use BertAdam in fp16 mode but the optimizer of NVIDIA's apex library.

OK thank you!