Weights of BertForQuestionAnswering not initialized from pretrained model
Author: ghost
Pull request: no

Trying to run cloned code from git but not able to train. Please suggest

`python run_squad.py   --bert_model bert-base-uncased   --do_train   --do_predict   --do_lower_case   --train_file $SQUAD_DIR/train-v1.1.json   --predict_file $SQUAD_DIR/dev-v1.1_sand.json   --train_batch_size 12   --learning_rate 3e-5   --num_train_epochs 2.0   --max_seq_length 384   --doc_stride 128   --output_dir debug_squad9`

I am constantly getting this error when run in training mode:

 ```
01/03/2019 12:22:39 - INFO - pytorch_pretrained_bert.modeling -   Weights of BertForQuestionAnswering not initialized from pretrained model: ['qa_outputs.weight', 'qa_outputs.bias']
01/03/2019 12:22:39 - INFO - pytorch_pretrained_bert.modeling -   Weights from pretrained model not used in BertForQuestionAnswering: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']
hon3.6/site-packages/torch/nn/functional.py", line 749, in dropout
    else _VF.dropout(input, p, training))
RuntimeError: $ Torch: not enough memory: you tried to allocate 0GB. Buy new RAM! at /pytorch/aten/src/TH/THGeneral.cpp:201
```

Comments:
What kind of GPU are you using?

I am on CPU as of now

```
(berttorch363) sandeepbhutani304@pytorch-bert-2:~/pytorch-pretrained-BERT/examples$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                2
On-line CPU(s) list:   0,1
Thread(s) per core:    2
Core(s) per socket:    1
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) CPU @ 2.00GHz
Stepping:              3
CPU MHz:               2000.146
BogoMIPS:              4000.29
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              56320K
NUMA node0 CPU(s):     0,1
```

I don't think it is possible to use BERT on CPU (didn't work for me). The model is too big.
If you find a way, feel free to re-open the issue.

Thanks for confirmation. Is "Fine Tuning" training also not possible for "BERT BASE"  on CPU? 
Correct me if I am wrong, following command is doing fine tuning training only
`python run_squad.py --bert_model bert-base-uncased --do_train --do_predict --do_lower_case --train_file $SQUAD_DIR/train-v1.1.json --predict_file $SQUAD_DIR/dev-v1.1_sand.json --train_batch_size 12 --learning_rate 3e-5 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir debug_squad9`

You are right.
And no, unfortunately, fine-tuning is not possible on CPU in my opinion.

When I use NVIDIA Corporation GM200 [GeForce GTX TITAN X], I also have this problem



Sorry - why is the model "too big" to be trained on CPU? Shouldn't the memory requirements of the GPU and CPU be basically the same? As far as I can tell, BERT should run on 12GB GPUs, plenty of CPU machines have more RAM than that. Or is there a difference in how the model is materialized in memory between CPU and GPU training?

My first comment was badly worded. You can `run` the model on CPU but `training` it on CPU is unrealistic.

Can you elaborate? My machine has 30gb of ram but indeed I've found I am running out of memory. How come a 12gb gpu is enough, what makes the difference? Also I'm talking about fine-tuning a multiple choice model, just for context.

How big your data @phdowling ? I have TPU for training, if you want I will give access to you