noamaanMulla-03/transformers-issues huggingface dataset:
`train_dataset` and `eval_dataset` in run_openai_gpt.py
Author: bkj
Pull request: no

Running `examples/run_openai_gpt.py` w/ the default arguments throws an error:
```
$ python run_openai_gpt.py --output_dir tmp --do_eval
Traceback (most recent call last):
  File "run_openai_gpt.py", line 259, in <module>
    main()
  File "run_openai_gpt.py", line 153, in main
    train_dataset = load_rocstories_dataset(args.train_dataset)
  File "run_openai_gpt.py", line 49, in load_rocstories_dataset
    with open(dataset_path, encoding='utf_8') as f:
FileNotFoundError: [Errno 2] No such file or directory: ''
```

Looking at the code, it looks like `train_dataset` and `eval_dataset` need to be explicitly set.  Any suggestions on what they should be set to?

Thanks!

cc @thomwolf 

Comments:
Hi Ben,
Please read the [relevant example section in the readme](https://github.com/huggingface/pytorch-pretrained-BERT#fine-tuning-openai-gpt-on-the-rocstories-dataset).

@thomwolf I see that the data is downloaded and cached in case of not providing the `train_dataset` and `eval_dataset` parameters: https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_openai_gpt.py#L152-L153, but it fails here: https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_openai_gpt.py#L162. So either we make the parameters required or do some additional steps which requires untar of the downloaded data and pointing towards that data. Let me know what you think and I can send a PR.
unicode
Author: elensergwork
Pull request: no

The general run_squad.py doesn't appear to work properly for python 2.7 because of the json  dumping  string vs unicode issues during the eval.  

python2.7 run_squad.py \
  --bert_model bert-base-uncased \
  --do_train \
  --do_predict \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /pytorch-pretrained-BERT/tmp/debug_squad/



02/15/2019 04:55:17 - INFO - __main__ -   Writing predictions to: /pytorch-pretrained-BERT/tmp/debug_squad/predictions.json
02/15/2019 04:55:17 - INFO - __main__ -   Writing nbest to: /pytorch-pretrained-BERT/tmp/debug_squad/nbest_predictions.json
Traceback (most recent call last):
  File "run_squad.py", line 1077, in <module>
    main()
  File "run_squad.py", line 1073, in main
    args.version_2_with_negative, args.null_score_diff_threshold)
  File "run_squad.py", line 619, in write_predictions
    writer.write(json.dumps(all_predictions, indent=4) + "\n")
TypeError: write() argument 1 must be unicode, not str


Comments:
Yes, the examples are not adapted for Python 2, only the library.
I don't plan to adapt or maintain them but feel free to submit a PR!

env: python2.7
line 662: writer.write(json.dumps(all_predictions, indent=4) + "\n")
change as :writer.write(json.dumps(all_predictions, indent=4).decode('utf-8') + "\n")
Pooler weights not being updated for Multiple Choice models? 
Author: meetps
Pull request: no

I'm trying use pretrained BERT to finetune on a multiple choice dataset.

The parameters from `pooler` are excluded from the optimizer params [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/modeling.py#L1044-L1047), however, the MutlipleChoice model does indeed use `pooled_output` (which passes through the `pooler`) [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_swag.py#L392).

I wasn't able to find a similar exclusion of `pooler` params from the optimizer in the official repo. I think I'm missing something here. Thanks for your patience.

Comments:
Indeed this looks like a bug in the `run_swag.py` example.
What do you think @rodgzilla?
Isn't the exclusion of the pooler parameters from optimization ([line 392 of `run_swag.py`](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_swag.py#L392)) a typo?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@thomwolf -  This surely looks like a bug, should I just send a hotfix and close it?

Yes!

Fixed in #675.
Can't find gpt2 vocab file. 
Author: suchithtuple
Pull request: no

When I run this 
```
tokenizer = GPT2Tokenizer.from_pretrained(pretrained_model_name_or_path='gpt2',cache_dir=None)
```
I am getting this 
```
Model name 'gpt2' was not found in model name list (gpt2). We assumed 'gpt2' was a path or url but couldn't find files https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json and https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt at this path or url.
```

Can anyone tell me where I am doing wrong. 

Comments:
I got the solution.
How can I generate new text after having fine-tuned BERT on a custom dataset ? 
Author: MoMe36
Pull request: no
Labels: name: Discussion

name: wontfix

Hey, 

Once I've fine-tuned the Language Model, how can I get it to generate new text ? Is there any example available ? 
Thanks !

Comments:
Also interested in this! 

Hi,
It's quite difficult to use BERT to generate text as BERT is not a causal language model per se.
Here is an example: https://github.com/nyu-dl/bert-gen by @W4ngatang and @kyunghyuncho.

Bert was not trained for text generation since it's not trained in the classical lm setting. However there are some new approaches that doesn't rely on next word predictions in the classical lm way. Have a look at: [Insertion Transformer](https://arxiv.org/abs/1902.03249) and [Insertion-based Decoding](https://arxiv.org/abs/1902.01370).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
BertForQuestionAnswering: How to split  output between query hidden state and context hidden state
Author: julietokwara
Pull request: no
Labels: name: Help wanted

name: Discussion

name: wontfix

I've made several attempts to, but all seem to fail. Do you have a good way to do this? Right now, passing what i thought to just be the context hidden state to the final output layer in run_squad.py drops my scores (F1) by 10 points. 

Comments:
If you want only the context you can find the index from the segment vector by finding the last first 1 in the vector and splitting the query_context on that index. Then the context will be everything after the 1 index and everything before will be the question.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Fixing issue "Training beyond specified 't_total' steps with schedule 'warmup_linear'" reported in #556
Author: samuelbroscheit
Pull request: yes

Fixing the issues reported in https://github.com/huggingface/pytorch-pretrained-BERT/issues/556

Reason for issue was that num_optimzation_steps was computed from example size, which is different from actual size of dataloader when an example is chunked into multiple instances.

Solution in this pull request is to compute num_optimization_steps directly from len(data_loader).

Comments:
Looks good.

Ok merging thanks, sorry for the delay!
Why the output of models are random.
Author: sjcfr
Pull request: no

I tried to get word representations using the full-retrained bert model for several times, whereas the outputs of model are different for a same word in each time. Did I neglect something? Not knowing the reason and asking for help sincerely.

The code is:
`from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMaskedLM `
`model = BertModel.from_pretrained('bert-base-uncased')`
`x=torch.LongTensor([[6541]])`
`y0=model(x)[0]`
`y1=model(x)[0]`

In theory, y0 should be equal to y1. However, they are different.

Both the length of y0 and y1 is 12, in accordance to the 12 layers of 'bert-base-uncased' model. However, each 12 element of y0 and y1 are different.  



Comments:
They won't be able to help you if you don't provide a code for reproducing your issue,  as this is not an expected behaviour.

Thanks a lot for reminding.  The issue is renewed with the code.

That's true! I can reproduce it also on my computer... Really weird!

You should use `model.eval()` to desactivate dropout like in the usage examples of the readme.

It solves the problem, thanks!
GPT-2 medium and large release?
Author: g-karthik
Pull request: no

I presume the below model is GPT-2 small.

https://github.com/huggingface/pytorch-pretrained-BERT/blob/ee0308f79ded65dac82c53dfb03e9ff7f06aeee4/pytorch_pretrained_bert/modeling_gpt2.py#L42

When do you plan on supporting the medium (already released by OpenAI) and large versions (not released by OpenAI) of GPT-2?

Thanks!

Comments:
Take a look at the `attention` branch @g-karthik:

https://github.com/huggingface/pytorch-pretrained-BERT/blob/attention/pytorch_pretrained_bert/modeling_gpt2.py#L42-L45

Thanks @julien-c, I had not looked at the file in the `attention` branch!

What is the recommended hardware setup for fine-tuning GPT2 medium?
Transformer XL ProjectedAdaptiveLogSoftmax bug (maybe?)
Author: shashwath94
Pull request: no

In <a href="https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/modeling_transfo_xl_utilities.py#L120">this line</a>, shouldn't the output be assigned to `out` when `n_clusters` is 0? Otherwise we run into `UnboundLocalError: local variable 'out' referenced before assignment`

Comments:
Yes! We don't see that when we use the pre-trained model because the number of clusters is greater than zero anyway. Will fix.

Thank you. I created a PR since it was a small bug. #690
Fix possible risks of bpe on special tokens
Author: SivilTaram
Pull request: yes
Labels: name: wontfix

Hi developers !

When I use the openai tokenizer, I find it hard to handle the `special tokens` correctly (my library version is v0.6.1) , even though I have already defined them and told the tokenizer NEVER SPLIT them. It is because all tokens, including the special ones will be processed by BPE. So I add one line for avoiding BPE on special tokens.

But there still are some problems when we use `spacy` as the tokenizer. I will try to add special tokens to the vocabulary of `spacy` and pull another request. Thanks for code review :)

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Limit on the input text length?
Author: lcswillems
Pull request: no

Hi,

I often get this error:

```
File "/miniconda3/envs/brightwater/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 268, in forward
    position_embeddings = self.position_embeddings(position_ids)
  File "/miniconda3/envs/brightwater/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/miniconda3/envs/brightwater/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 117, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/miniconda3/envs/brightwater/lib/python3.6/site-packages/torch/nn/functional.py", line 1506, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range at ../aten/src/TH/generic/THTensorEvenMoreMath.cpp:193
```

It only happens for long texts. It doesn't fail on chunks of a long text that is failing.

Is there a limitation on the length of the input text?

Comments:
Yes, 512 tokens for Bert.

Thank you :) 

Is there a way to bypass this limit? To increase the number of words?
Understanding pre-training and fine-tuning
Author: BramVanroy
Pull request: no
Labels: name: Discussion

name: wontfix

I am confused about what these two steps actually do to the model. I would have assumed that pre-training is unsupervised (i.e. no labels) and, thus, the only thing that can be 'learned' is the embedding representations of all tokens. You can then use this pre-trained model (which is an 'empty' model but with pretrained vector representations of tokens) to actually train the whole mode.

However, from the things that I read that is not the case. It's not only the embeddings that are pretrained, but also the whole model. Is that right? How can that be, without labels?

For context: I want to train a Bert model on a new language and I am trying to figure out which steps I have to do to get there. Can I load custom embeddings, and do I then still need to pretrain the rest of the model (but how, which labels?).

I know that my question is quite general, but the difference between pre-training and fine-tuning is not clear to me. Pre-training should be 'done once per language', but then what does the model actually learn? It it simply a seq-to-seq pre-training?

Comments:
Maybe a good introduction on the topic are the writings of @sebastianruder:
- http://ruder.io/transfer-learning/
- https://thegradient.pub/nlp-imagenet/
- the ULMFiT paper: http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html

Regarding your specific question of training a Bert model on a new language, the computing requirements for training BERT are very high. You would probably be better with a version of ULMFiT. There is a huge multilingual initiative on the fast.ai forum that you can check out. One entry point is [here](https://forums.fast.ai/t/language-model-zoo-gorilla/14623).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
BERT  what's different with step and t_total
Author: xuanzebi
Pull request: no

 :param t_total: how many training steps (updates) are planned 
:param step:    which of t_total steps we're on


def get_lr(self, step, nowarn=False):
        """
        :param step:    which of t_total steps we're on
        :param nowarn:  set to True to suppress warning regarding training beyond specified 't_total' steps
        :return:        learning rate multiplier for current update
        """
        if self.t_total < 0:
            return 1.
        progress = float(step) / self.t_total
        ret = self.get_lr_(progress)
        # warning for exceeding t_total (only active with warmup_linear
        if not nowarn and self.warn_t_total and progress > 1. and progress > self.warned_for_t_total_at_progress:
            logger.warning(
                "Training beyond specified 't_total'. Learning rate multiplier set to {}. Please set 't_total' of {} correctly."
                    .format(ret, self.__class__.__name__))
            self.warned_for_t_total_at_progress = progress
        # end warning
        return ret

Comments:
How to incrementally do fine tune train 
Author: shuvadibp
Pull request: no
Labels: name: Discussion

name: wontfix

I am using Bert for Question Answering. After fine tuning with Squad data set, I want to further train new questions of my own domain.   
Please suggest how can I use newly generated pytorch_model.bin file and then increment it with my own training weights to get my own pytorch_suqad_plus_my_model.bin ?

Comments:
I think my answer here can help you: https://github.com/huggingface/pytorch-pretrained-BERT/issues/332

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hey @shuvadibp, did you figure out a way of doing it? I would like to talk to you about the same..
Have no GPU to train language modelling
Author: khaerulumam42
Pull request: no

Sorry I open this issue, is not issue of this repository. 

I very appreciate what the authors created this repository, help us to more understand how BERT works and implement on several tasks.

So I have a problem with training because I have not GPU to train language modelling, I have Indonesian dataset (about 2GB) that trainable for language modelling using this repo, could anyone help me to train this dataset? If you could help, you have permission to open source or use the model trained. 

I hope it will be give more models provided and to make NLP community more interest in latest NLP models especially Indonesian.

You can email me directly on khaerulumam42@gmail.com or comment below.

Thank you very much

Comments:
[You can train a tensorflow model using google colab for free](https://github.com/google-research/bert#using-bert-in-colab). After training it, you can [convert your tf model to pytorch](https://github.com/huggingface/pytorch-pretrained-BERT#command-line-interface). 

Or use 300 usd credit for google cloud, that you get when you signup i believe.

Thank you @oliverguhr and @Oxi84 for suggestions. 

I have tried both methods, using google colabs and GPU as runtime processor, it took about 240hours for every epoch (maybe if I use apex, it will be faster but I think still hundreds of hours), i think it's impossible to run google colabs dozens of days.

I got free trial for GCP, unfortunately Google not provide GPU for free trial version. I try training use GCP with 2CPU and 13GB RAM, it take 200thousands of hours training, is sooo long time.

Maybe I should reduce corpus size?

Thanks

Or smaller vocabulary. I am pretty sure you can even use TPU on Google cloud, let someone else confirm that.

@Oxi84 
For my classification task, I noticed that training the model with just 40 mb of data will give me already pretty good results. Training with the full 1,5 GB of my dataset improves the results by just 2-3% accuracy. 
So you might start with a (random)  subset of your data and improve the size step by step and see if your scores get better. 

Oh nice insight @oliverguhr , thank you. I will try to reduce training data and train.
Pregenerating data requires multiple documents
Author: yanneyanne
Pull request: no
Labels: name: Discussion

name: wontfix

The script for pregenerating language modelling data assumes that the training corpus consists of multiple documents (i.e. a single training corpus file where empty lines separate documents). If the training corpus is made up of only one long text, the pregen script produces empty output.

I have a small fix for this locally, which enables pregenerating from a single document. I could create a PR if the empty output is not seen as expected behaviour.

(I can also mention that the **Installation** section of the README states that the repo has been tested with python 3.5+. However, the pregen script uses f-strings, which are a python 3.6 feature.)

Comments:
Hi, I wrote that script! That's a tricky issue, though - what behaviour do you expect when your data is one long text? 

In the original BERT repo, they used the document breaks to control sampling for the NextSentence task - 'random' next sentences were selected from a different document. I'm not sure there's a "general" solution to the case when all the data is a single document, though - people whose data is all one contiguous document will probably have different expectations for how they want that data to be handled (e.g. is it okay to select a 'random' sentence 10 sentences away, or should it be much further? Are 'nearby' sentences too similar?)

I suppose we could add code to support that with a controllable parameter to stop it sampling random sentences that are too close to the actual sentence, but it might be easier just to throw an error if the data is only one single document and request that the user break it into 'documents' based on whatever structure is in the data, like chapter or page breaks, or in the worst case just add a break every 1,000 lines or something.

You're totally right about the f-strings, though. I'll look into updating the README!

I'm assuming that the script should just grab any random sentence in the same (only) document. I could see how that particular behavior might not be desired however.

The error-throwing idea is good. I ran the script on my mammoth document, only to see empty output after 9 hours (my own fault obviously, but an exception might help similarly sloppy people).

Added PR #478 to address this. Unfortunately, it probably still won't resolve your 9 hour problem - the script will only realize there was only one document in the corpus after it's finished reading and tokenizing the whole thing! Still, at least people will get an error message properly explaining the issue and suggesting fixes now.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Difference between base and large tokenizer?
Author: ZhaofengWu
Pull request: no
Labels: name: Discussion

name: wontfix

I understand that a cased tokenizer and an uncased one are surely different because their vocabs are different in casing, but how does a base tokenizer different from a large tokenizer? Does a large tokenizer have a larger vocab?

Comments:
I haven't looked in the details of the vocabularies for each model.
If you investigate this question, be sure to share the results here, it may interest others as well!

I did a diff on the two vocabulary files and there is no difference. As long as you use the uncased version at least. I haven't investigated others.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I can validate @mhattingpete 's research.
I tokenized a big collection of text with the uncased tokenizer from both the base and the large model and both tokenizations are identical.
Add an argument --model_size to convert_gpt2_checkpoint_to_pytorch.py
Author: chrisyue1998
Pull request: yes

Add an argument --model_size to convert_gpt2_checkpoint_to_pytorch.py that lets the user specify whether they want to convert a checkpoint from the 117M model or from the 345M model so that they don't have to create their own 345M json config file.

Comments:
FileNotFoundError: [Errno 2] No such file or directory: 'uncased_L-12_H-768_A-12\\pytorch_model.bin'
Author: DEBADRIBASAK
Pull request: no

I was just trying to get familiar with the pytorch implementation of BERT. I tried with the examles mentioned in the README file. The statement : **tokenizer = BertTokenizer.from_pretrained(BERT_PRETRAINED_PATH,do_lower_case=True)** works perfectly but when I try the same with **model = BertForMaskedLM.from_pretrained(BERT_PRETRAINED_PATH)**, it shows the error : **FileNotFoundError: [Errno 2] No such file or directory: 'uncased_L-12_H-768_A-12\\pytorch_model.bin'**   Can someone point out where I am going wrong? The pretrained bert weights are available in the same directory and the **BERT_PRETRAINED_PATH** is **Path("uncased_L-12_H-768_A-12")**

Comments:
I got the same problem. Did you solve the problem?

> I got the same problem. Did you solve the problem?

Yes. Actually the project folder of this implementation does not contain the `pytorch_model.bin` file. For loading the actual pretrained model, you have to use `BertModel.from_pretrained('bert-base-uncased').` Here as the parameter, you can mention name of any one of the bert variations available. After fine-tuning, you can save the `state_dict()` of the model in the `pytorch_model.bin` file and use it later.