noamaanMulla-03/transformers-issues huggingface dataset:
fix prediction on run-squad.py example
Author: JoeDumoulin
Pull request: yes

run_squad.py exits with an error when running do_predict without training.  The error is due to the model_state_dict not existing when --do_predict is selected.

Traceback (most recent call last):
  File "run_squad.py", line 980, in <module>
    main()
  File "run_squad.py", line 923, in main
    model_state_dict = torch.load(output_model_file)
  File "/home/joe/pytorchnlp/lib/python3.5/site-packages/torch/serialization.py", line 365, in load
    f = open(f, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '../../BERT_work/Squad/pytorch_model.bin'

Comments:
Thanks @JoeDumoulin
Fix squad answer start and end position
Author: cooelf
Pull request: yes

Previous version might miss some overlong answer start and end indices (which should be 0), and sometimes the start/end positions would be outside the model inputs.

Doc_start and doc_end are based on tokenized subword sequences while example.start_position and example.end_position are indices in original word-level, which are usually shorter than the subword sequences. Why not use tok_start_position and tok_end_position for comparison?

Comments:
Thanks @cooelf, there was a PR merging `run_squad` and `run_squad2` that also fixed this issue.
What is get_lr() meaning in the optimizer.py
Author: kugwzk
Pull request: no

I use a Model based on BertModel, and when I use the BertAdam the learning rate isn't changed. And when I use `get_lr()`, the return result is `[0]`. And I see the length of state isn't 0, but why I get that?

Comments:
You can use it to get the current learning rate of the `BertAdam` optimizer (which vary according to the schedules discussed in #195).
BERT tuning all parameters? 
Author: danyaljj
Pull request: no

Just a clarification question: 
when tuning bert parameters (say, for SQUAD), does it tune the parameters of the final parameter or the whole BERT model? 


Comments:
In the examples scripts, it tunes the whole model.
But BERT models classes are just regular PyTorch `nn.Modules` so you can also freeze layer like you would do in any PyTorch module.
Fine tuning Bert for Question answering 
Author: MathewAlexander
Pull request: no

I just wanted to ask if the weights of the Bert base model are getting updated while fine tuning Bert for Question answering. I see that the  Bert for QA is a model with A linear layer on top of Bert pre-trained model. I am trying to reproduce the same model in keras. Could any one tell me if i should freeze the layers in the Bert base model or not?


Comments:
The weights of the BERT base model are getting updated while finetuning the network.

Indeed
can you do a new release + pypi
Author: joelgrus
Pull request: no

we've been getting some requests to incorporate newer features into allennlp that are only on master (e.g. `never_split`).

thanks!

Comments:
Hi Joel, yes the new release (0.5.0) is pretty much ready (remaining work on branches `fifth-release` and `transfo-xl` to finish testing the newly added pre-trained OpenAI GPT and Transformer-XL).

Likely next week.

Awesome, thanks!

On Thu, Jan 31, 2019, 11:06 AM Thomas Wolf <notifications@github.com> wrote:

> Hi Joel, yes the new release (0.5.0) is pretty much ready (remaining work
> on branches fifth-release and transfo-xl to finish testing the newly
> added pre-trained OpenAI GPT and Transformer-XL).
>
> Likely next week.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/pytorch-pretrained-BERT/issues/245#issuecomment-459506373>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABP2mYQxXrEKfGvJhESKqKIQzI2dvcolks5vI1rUgaJpZM4ac94Y>
> .
>


Ok @joelgrus, the new release is out: https://github.com/huggingface/pytorch-pretrained-BERT/releases/tag/v0.5.0
potential bug in extract_features.py
Author: jayleicn
Pull request: no

Hi,

`token_type_ids` is not set for this line: 
`all_encoder_layers, _ = model(input_ids, token_type_ids=None, attention_mask=input_mask)`
 https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/extract_features.py#L267, this does not affect single sequence feature extraction, but for a pair of sequence, the model will process the pair as a single sequence and add `A` embedding to the two sequences, which should add `A`, `B` respectively. Seems like a bug.

Best,
Jie

Comments:
Hi Jie,
`extract_feature.py` is an example script. If you want to adapt it for sentences-pair, we would be happy to welcome a PR :)
Missing files for Transformer-XL examples
Author: stefan-it
Pull request: no

Hi,

thanks so much for the new *0.5.0* release. I wanted to train a `TransfoXLModel` model, as described in the `README` [here](https://github.com/huggingface/pytorch-pretrained-BERT/blame/master/README.md#L132).

Unfortunately, the files `transfo_xl_train.py` and `transfo_xl_eval.py` are not located in the `examples` directory.

Could you please add them to repository? Thanks :heart: 

Comments:
Oh yes, that was a typo, there is only one example for Transformer-XL and it's the `run_transfo_xl.py` file detailed [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/README.md#openai-gpt-and-transformer-xl-running-the-examples). Read the API details in the readme for more information on the input/outputs of the two Transformer-XL models.
I've updated the readme, thanks.
Variance Sources
Author: carolinlawrence
Pull request: no

Hi,

when I change the `--seed` argument, I get a high variance between different runs on my dataset. So I was wondering where the sources of variance might come from. I see that the seed is set (e.g. in `run_squad.py`) via:

`random.seed(args.seed)
np.random.seed(args.seed)
torch.manual_seed(args.seed)`

But how can I find out where randomization is actually used?

I found the `RandomSampler` and replaced it with a `SequentialSampler`, but the variance remains high.

I know that `modeling.py` randomly initializes the weights, but these are overwritten by the fixed weights when loading a pre-trained BERT model, e.g. `bert-base-uncased`, correct?

Can anyone point me in any other direction where my source of variance might come from?

Thanks!

Comments:
Hi Carolin,

Depending on the model you are using, not all the weights are initialized from the pre-trained models. Check the details in the [overview](https://github.com/huggingface/pytorch-pretrained-BERT#overview) section of the readme to see if it's the case for you.

Apart from weights initialization and dataset shuffling other typical source of variances are the dropout layers.

Bert fine-tuning has been reported to be a high-variance process indeed, in particular on small datasets.

Hi Thomas,

thanks for the quick reply!

I'm using `BertForMaskedLM`, so the weights should be set. But yes, I didn't think of dropout, thanks for pointing that out!
Argument do_lower_case is repeated in run_lm_finetuning.py
Author: dileep1996
Pull request: no

Hi, I am trying to finetune LM and am facing the following issue. 

**argparse.ArgumentError: argument --do_lower_case: conflicting option string: --do_lower_case**

Comments:
Hi @dileep1996, this has just been fixed in master (#275)!
max sentence length
Author: RayXu14
Pull request: no

is there an max sentence length for this bert code?

Comments:
Hi, 512 tokens if you use the pre-trained models. Any length you want if you train your models from scratch.

could we set it smaller ? cause if i set it as 512, then result is out of memory

You can just send a smaller input in the model, no need to go to the max

thank you @thomwolf
Get hidden states from all layers of Transformer-XL?
Author: hugochan
Pull request: no

Hi,

Thank you for supporting the pretrained Transformer-XL model! I was wondering if it makes sense to get hidden states from all layers of Transformer-XL as the output, just as what can be done for BERT. It seems this is not supported currently. Practically I found this strategy worked well for BERT and gave better results. Not sure if it is a good idea for Transformer-XL. Thank you!

Comments:
Hi @hugochan, actually that what's in the `mems` of the Transformer-XL are (maybe you can read again the paper).

One thing to be careful about is that the `mems` have transposed first dimensions and are longer (see the readme). Here is how to extract the hidden states from the model output:
```python
hidden_states, mems = model(tokens_tensor)
seq_length = hidden_states.size(1)
lower_hidden_states = list(t[-seq_length:, ...].transpose(0, 1) for t in mems)
all_hidden_states = lower_hidden_states + [hidden_states]
```

Hi @thomwolf , thank you for your answer! Just one quick question. It seems that `mems` already contains a list of num_layer hidden states, what is the difference between `lower_hidden_states[-1]` and `hidden_states` in your code? Thank you!

Actually `mems` contains all the hidden states PLUS the output of the embeddings (`lower_hidden_states[0]`) so `lower_hidden_states[-1]` is the output of the hidden state of the layer below the last layer and `hidden_states` is the output of the last layer (before the softmax).

I will add a note on that in the readme.
Added Squad 2.0
Author: abeljim
Pull request: yes

Accidentally closed the last pull request. Created Separate file for Squad 2.0 run with 
python3 run_squad.py 
--bert_model bert-large-uncased_up 
--do_predict 
--do_lower_case 
--train_file squad/train-v2.0.json 
--predict_file squad/dev-v2.0.json 
--learning_rate 3e-5 
--num_train_epochs 2 
--max_seq_length 384 
--doc_stride 128 
--output_dir squad2_diff 
--train_batch_size 24 
--fp16 
--loss_scale 128 
--null_score_diff_threshold -2.6929588317871094
If null score not defined default value is 0.0

Comments:
Great, thanks @abeljim.
Do you have the associated results when you run this command?

python run_squad2.py \
  --bert_model bert-large-uncased \
  --do_train \
  --do_predict \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v2.0.json \
  --predict_file $SQUAD_DIR/dev-v2.0.json \
  --train_batch_size 24 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir ./debug_squad/ \
  --fp16 \
  --loss_scale 128 \
  --null_score_diff_threshold -2.6929588317871094

{
  "exact": 76.62764255032427,
  "f1": 79.22523967450329,
  "total": 11873,
  "HasAns_exact": 68.31983805668017,
  "HasAns_f1": 73.52248155455082,
  "HasAns_total": 5928,
  "NoAns_exact": 84.9116904962153,
  "NoAns_f1": 84.9116904962153,
  "NoAns_total": 5945
}
This is the command I used and the results

how much time does it take to train?
Fix some bug about SQuAD code
Author: wlhgtc
Pull request: yes

Fix issue in #207 
![image](https://user-images.githubusercontent.com/16603773/52842280-90939a00-3139-11e9-87e8-92bcfb8af76d.png)

This error occurs when 'nbest' only contain 1 item, but none 'text'. So the code to add empty will not work.
I add another condition to solve it.

https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_squad.py#L570-L590

Comments:
Ok, thanks @wlhgtc!
bert.pooler.dense initialization 
Author: SinghJasdeep
Pull request: no

Hey! Sorry if this is redundant. 

I saw 3 other issues asking similar questions but couldn't find these exact layers mentioned. 

Are bert.pooler.dense.weight  &  bert.pooler.dense.bias randomly initialized? 

Thank you so much! 


Comments:
Hi Jasdeep,
No, they are initialized from Google's pretrained model (they are trained for next sentence prediction task during pretraining).
Example Code in README fails.
Author: blester125
Pull request: no

There is an assertion error in the example code in the README.

The text that  is input to the model 

`"[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"`

is expected to be tokenized and masked like so

`['[CLS]', 'who', 'was', 'jim', 'henson', '?', '[SEP]', 'jim', '[MASK]', 'was', 'a', 'puppet', '##eer', '[SEP]']`. 

However the `[CLS]` and `[SEP]` tokens are split up and in the class of `[CLS]` it is broken into word pieces. The actual tokenized results looks like this

`['[', 'cl', '##s', ']', 'who', 'was', 'jim', 'henson', '[MASK]', '[', 'sep', ']', 'jim', 'henson', 'was', 'a', 'puppet', '##eer', '[', 'sep', ']']`

Comments:
This could be related to #266 - are you using the latest version of `pytorch-pretrained-BERT`?

No, I was on 0.4, I upgraded to 0.6.1 and it worked.
run_squad.py stuck on batch size greater than 1
Author: wcgan
Pull request: no

Thanks a lot for the code! I need help figuring out why the script is not working so long the batch_size is set to be above 1. Specifically, it seems to be stuck at Line 908: loss = model(input_ids, segment_ids, input_mask, start_positions, end_positions). I am using 4 k80. Thanks!

Comments:
Please copy paste the command you are using to run this example.

Here you go

```
python ./run_squad.py 
  --bert_model bert-base-uncased \
  --do_train \
  --do_predict \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --learning_rate 3e-5 \
  --num_train_epochs 2 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --train_batch_size 32 \
  --output_dir /tmp/debug_squad/ \
  --gradient_accumulation_steps 2 \
```

I don't see why this wouldn't work.
Maybe update the repo & module to the latest version?
You should also add `--do_lower_case` to the arguments if you are using an uncased model.
Maybe post a full log of your output?

Updated to the latest version but it still does not work. When I terminate the script after it is stuck for some time, I get the message  '../python3.7/threading.py", line 1072, in _wait_for_tstate_lock    elif lock.acquire(block, timeout)'. Perhaps it is running into some deadlock condition?

I'm not sure how to obtain a full log, would you be able to explain how can I do so? Thanks!

Hi, you can try with the new release 0.4.0.

Is the problem resolved? I am having the same issue, using 2 gtx1080ti. 
it stuck when running on multiple gpus. I have to comment out `torch.nn.DataParallel(model)`, to make it work. 

If you are using a multi-GPU setting, pytorch splits the batch dynamically between the 2 GPUs.
Example - batch_size =5
GPU 0 may get 3,max_sequence_len
GPU 1 may get 2,max_sequence_len

This could be a cuda splitting issue, I recommend you try a single GPU setting to debug this.

Thanks,
Ankit
Can I do a code reference in implementing my code?
Author: graykode
Pull request: no

@thomwolf 
I am trying simply implementing gpt-2 on Pytorch
I have trouble in trasfering tensorflow checkpoint to pytorch :(
https://github.com/graykode/gpt-2-Pytorch
Could I do this code reference in implementing my code? I'll write reference in my code!!
Thanks for awesome sharing!

Comments:
Hi @graykode,
What do you mean by "code reference"?

@thomwolf 
Hello thomwolf!
It mean that I apply your code about `GPT-2 model and transferring tensorflow checkpoint to pytorch` in my project!
 I show the origin of the information in my project code comment when I refer to your code.
Thanks

Oh yes, no problem.
Just reference the origin of the work and the licences (inherited from the relevant authors and code I started from)

@thomwolf Sure, What license i follow? I already known original [openAi/gpt-2](https://github.com/openai/gpt-2) is MIT license, but pytorch-pretrained-BERT is Apache 2.0!
I want to just use gpt-2 model and model transferring code!!
Issue with apex import on MAC 
Author: bhoomit
Pull request: no

Python 3.7
MacOS High Sierra 10.13.6

```
Traceback (most recent call last):
  File "examples/classifier.py", line 1, in <module>
    from pytorch_pretrained_bert.tokenization import BertTokenizer, WordpieceTokenizer
  File "/Users/Bhoomit/work/robin/nlp/pytorch-pretrained-BERT/env/lib/python3.7/site-packages/pytorch_pretrained_bert/__init__.py", line 7, in <module>
    from .modeling import (BertConfig, BertModel, BertForPreTraining,
  File "/Users/Bhoomit/work/robin/nlp/pytorch-pretrained-BERT/env/lib/python3.7/site-packages/pytorch_pretrained_bert/modeling.py", line 218, in <module>
    from apex.normalization.fused_layer_norm import FusedLayerNorm as BertLayerNorm
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/Users/Bhoomit/work/robin/nlp/pytorch-pretrained-BERT/env/lib/python3.7/site-packages/apex-0.1-py3.7.egg/apex/__init__.py", line 12, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/Users/Bhoomit/work/robin/nlp/pytorch-pretrained-BERT/env/lib/python3.7/site-packages/apex-0.1-py3.7.egg/apex/optimizers/__init__.py", line 2, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/Users/Bhoomit/work/robin/nlp/pytorch-pretrained-BERT/env/lib/python3.7/site-packages/apex-0.1-py3.7.egg/apex/optimizers/fp16_optimizer.py", line 8, in <module>
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ctypes/__init__.py", line 369, in __getattr__
    func = self.__getitem__(name)
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ctypes/__init__.py", line 374, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(RTLD_DEFAULT, THCudaHalfTensor_normall): symbol not found
```

Comments:
Unfortunately, apex (and fp16 in general) only work on GPU. So you can't use it on MacOS :/
will examples update the parameters of bert model?
Author: susht3
Pull request: no

on the examples, it loads bert-base model and do some tasks, the paper says that it will fix the parameters of bert and only update  the parameters of our tasks, but i find that it seems not fix parameters of bert? just load the model, and adds some layers to train

Comments:
Could you please show the part of the paper where you have seen mentioned, I haven't found it.

Are you talking about this paragraph?

>In this section we evaluate how well BERT performs in the feature-based approach by generating ELMo-like pre-trained contextual representations on the CoNLL-2003 NER task. To do this, we use the same input representation as in Section 4.3, but use the activations from one or more layers with- out fine-tuning any parameters of BERT. These contextual embeddings are used as input to a randomly initialized two-layer 768-dimensional BiLSTM before the classification layer.

Closing this since there no activity. Feel free to re-open if needed.

i think it is asking whether we are fine-tuning the whole bert model or use bert outputs as a fixed feature for representing the sentences (like ELMO)
anyone notice large difference of using fp16 ?
Author: howardhsu
Pull request: no

I recently noticed that using fp16 dropped the performance of BERT on my own dataset but improved on another (it works fine on examples like MPRC). It's about 4% so unlikely to be random noise.

I'm trying to see the reason and noticed examples from apex:
https://github.com/NVIDIA/apex/tree/master/examples
actually uses a global copy of the parameters during training but examples in this repository just use fp16 for all steps (and for saving parameters.) Wondering will this be the potential reason?

Comments:
update documentation for gpt-2
Author: joelgrus
Pull request: yes

fixes a few incorrect details in the gpt-2 documentation.

one remaining thing, all of the models return an extra `presents` variable that I'm not quite sure what it is, so there's a ... in the doc. if you tell me what to put there I can put it there, or you can do it yourself.

Comments:
Hi @joelgrus, you are right, the docstring were lagging a lot.
All the information is in the `README.py`, more specifically [these sections detailing the API of the GPT2 models](https://github.com/huggingface/pytorch-pretrained-BERT#14-gpt2model) but I forgot to update the docstrings. Do you want to have a look and tell me if it's detailed enough for the docstrings also?

yes, that's great, I made those changes. (I also feel kind of dumb for not looking at the docs docs.)

sorry about the trailing whitespace changes in the README.md, my editor removes those automatically.

Thanks Joel!
Checkpoints not saved
Author: ylhsieh
Pull request: no

There is an option `save_checkpoints_steps` that seems to control checkpointing. However, there is no actual saving operation in the `run_*` scripts. So, should we add that functionality or remove this argument?

Comments:
In the `run_squad.py`script, I added the following lines after the training loop:

```
logger.info(***** Saving fine-tuned model *****)
output_model_file = os.path.join(args.output_dir, "pytorch_model.bin")
if n_gpu > 1:
    torch.save(model.module.bert.state_dict(), output_model_file)
else:
    torch.save(model.bert.state_dict(), output_model_file)
```

The code runs and I was able to load the model to test on the Adversarial SQuAD datasets.

I do not use the other `run_*` scripts but this may be applicable as well.

Edit: the files have been modified in the latest commits so I think it's now necessary to check the loading of fine-tuned models in the script.

You are right this argument was not used. I removed it, thanks. These examples are provided as starting point to write training scripts for the package module. I don't plan to update them any further (except fixing bugs).

> In the `run_squad.py`script, I added the following lines after the training loop:
> 
> ```
> logger.info(***** Saving fine-tuned model *****)
> output_model_file = os.path.join(args.output_dir, "pytorch_model.bin")
> if n_gpu > 1:
>     torch.save(model.module.bert.state_dict(), output_model_file)
> else:
>     torch.save(model.bert.state_dict(), output_model_file)
> ```
> The code runs and I was able to load the model to test on the Adversarial SQuAD datasets.
> 
> I do not use the other `run_*` scripts but this may be applicable as well.
> 
> Edit: the files have been modified in the latest commits so I think it's now necessary to check the loading of fine-tuned models in the script.

what is your result on adversarial-squad?

At that time I got:
**AddSent**
BERT base 58.7 EM / 66.2 F1
BERT large 65.5 EM / 71.9 F1

**AddOneSent**
BERT base 67.0 EM / 74.7 F1
BERT large 72.7 EM / 79.1 F1





> At that time I got:
> **AddSent**
> BERT base 58.7 EM / 66.2 F1
> BERT large 65.5 EM / 71.9 F1
> 
> **AddOneSent**
> BERT base 67.0 EM / 74.7 F1
> BERT large 72.7 EM / 79.1 F1

Thanks a lot! Do you release your paper? i want to cite your result and paper in my paper.

Unfortunately it was not part of a paper, just preliminary results.