Model not training at all in Google Colab
Author: nicholasbailey87
Pull request: no
Labels: name: Help wanted

name: Discussion

name: wontfix

Hi! Thanks for your help.

I am initiating training in the following way in a Colab notebook with GPU acceleration (with very small train batch size and max seq length to prove I'm not getting out of memory problems!):

    !pip install pytorch-pretrained-bert
    !rm -rf bert_output
    !mkdir bert_output
    !python ./pytorch-pretrained-BERT/examples/run_squad.py --bert_model bert-base-multilingual-cased --output_dir bert_output --train_file squad_20_train.json --predict_file squad_20_dev.json --do_train --do_predict --train_batch_size 64 --max_seq_length 64 --version_2_with_negative --fp16

I have cloned the pytorch-pretrained-bert repository and installed the library. I have also installed apex. However, training doesn't even get started - the last thing the verbose logging produces is:

    04/01/2019 14:36:45 - INFO - __main__ -   tokens: [CLS] When did Bey ##once take a hi ##atus in her career and take control of her management ? [SEP] . Her critically acclaimed fifth studio album , Beyoncé ( 2013 ) , was distinguished from previous releases by its experimental production and exploration of dark ##er themes . [SEP]
    04/01/2019 14:36:45 - INFO - __main__ -   token_to_orig_map: 20:137 21:138 22:139 23:140 24:141 25:142 26:143 27:143 28:144 29:145 30:145 31:145 32:145 33:146 34:147 35:148 36:149 37:150 38:151 39:152 40:153 41:154 42:155 43:156 44:157 45:158 46:158 47:159 48:159
    04/01/2019 14:36:45 - INFO - __main__ -   token_is_max_context: 20:True 21:True 22:True 23:True 24:True 25:True 26:True 27:True 28:True 29:True 30:True 31:True 32:True 33:True 34:True 35:True 36:True 37:True 38:True 39:True 40:True 41:True 42:True 43:True 44:True 45:True 46:True 47:True 48:True
    04/01/2019 14:36:45 - INFO - __main__ -   input_ids: 101 12242 12172 40344 104316 13574 169 11520 26311 10106 10485 13021 10111 13574 12608 10108 10485 17150 136 102 119 13229 108889 87680 22237 13093 10606 117 54106 113 10207 114 117 10134 45233 10188 16741 45906 10155 10474 34176 12116 10111 61326 10108 25100 10165 48462 119 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    04/01/2019 14:36:45 - INFO - __main__ -   input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    04/01/2019 14:36:45 - INFO - __main__ -   segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    04/01/2019 14:36:45 - INFO - __main__ -   start_position: 0
    04/01/2019 14:36:45 - INFO - __main__ -   end_position: 0
    04/01/2019 14:36:45 - INFO - __main__ -   answer: [CLS]

So, it seems as though the notebook is preprocessing the data OK, but then it hangs here and ultimately terminates the process with a ^C (not from me, but something internal).

Any idea what the problem might be here?

Thanks again!

Nicholas

Comments:
Try train_batch_size =1. Alternatively I propose to finetune with the tensorflow model using colabs TPUs as these have far more memory.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.