cannot run squad script 
Author: jiahuigeng
Pull request: no
Labels: name: Need more information

name: wontfix

(g-torch) [bipkg@SVR16173HP380 examples]$ python run_squad.py --bert_model bert-base-uncased  --train_file squad/train-v1.1.json --do_train  --output_dir exps2
04/12/2019 11:32:03 - INFO - __main__ -   device: cuda n_gpu: 1, distributed training: False, 16-bits training: False
04/12/2019 11:32:03 - WARNING - pytorch_pretrained_bert.tokenization -   The pre-trained model you are loading is an uncased model but you have set `do_lower_case` to False. We are setting `do_lower_case=True` for you but you may want to check this behavior.
04/12/2019 11:32:05 - INFO - pytorch_pretrained_bert.tokenization -   loading vocabulary file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /home/bipkg/data/jgeng/Bert/pytorch_pretrained_bert/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
04/12/2019 11:32:16 - INFO - pytorch_pretrained_bert.modeling -   loading archive file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased.tar.gz from cache at /home/bipkg/data/jgeng/Bert/pytorch_pretrained_bert/distributed_-1/9c41111e2de84547a463fd39217199738d1e3deb72d4fec4399e6e241983c6f0.ae3cef932725ca7a30cdcb93fc6e09150a55e2a130ec7af63975a16c153ae2ba
04/12/2019 11:32:16 - INFO - pytorch_pretrained_bert.modeling -   extracting archive file /home/bipkg/.pytorch_pretrained_bert/distributed_-1/9c41111e2de84547a463fd39217199738d1e3deb72d4fec4399e6e241983c6f0.ae3cef932725ca7a30cdcb93fc6e09150a55e2a130ec7af63975a16c153ae2ba to temp dir /tmp/tmpj0z3ai2_
04/12/2019 11:32:20 - INFO - pytorch_pretrained_bert.modeling -   Model config {
  "attention_probs_dropout_prob": 0.1,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "max_position_embeddings": 512,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "type_vocab_size": 2,
  "vocab_size": 30522
}

04/12/2019 11:32:25 - INFO - pytorch_pretrained_bert.modeling -   Weights of BertForQuestionAnswering not initialized from pretrained model: ['qa_outputs.weight', 'qa_outputs.bias']
04/12/2019 11:32:25 - INFO - pytorch_pretrained_bert.modeling -   Weights from pretrained model not used in BertForQuestionAnswering: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']
04/12/2019 11:32:42 - INFO - __main__ -   ***** Running training *****
04/12/2019 11:32:42 - INFO - __main__ -     Num orig examples = 87599
04/12/2019 11:32:42 - INFO - __main__ -     Num split examples = 88641
04/12/2019 11:32:42 - INFO - __main__ -     Batch size = 32
04/12/2019 11:32:42 - INFO - __main__ -     Num steps = 8211
Epoch:   0%|                                                                                                                                                                                                                                             | 0/3 [00:00<?, ?it/s]Segmentation fault (core dumped)                                                                                                                                                                                                                      | 0/2771 [00:00<?, ?it/s]


Comments:
What hardware are you using?
What is your software configuration (versions of python, pytorch, pytorh-pretrained-bert, etc...)?

I'm not so sure that this is the same problem that I experienced.

If you installed apex, the problem may be related to BertLayerNorm, because BertLayerNorm will use FusedLayerNorm of apex library. Thus, if you disable to use FusedLayerNorm of apex library, then the segmentation fault will disappear.

The other solution is that you should reinstall apex. Please refer to https://github.com/NVIDIA/apex/issues/156#issuecomment-465301976.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@cartopy  reinstall apex works well thx

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.