Bert additional pre-training

maria: <p>I would like to use transformers/hugging face library to further pretrain BERT. I found the masked LM/ pretrain model, and a usage <a href="https://huggingface.co/transformers/model_doc/bert.html#bertforpretraining" rel="nofollow noopener">example</a>, but not a training example.<br>
In the original BERT repo I have this <a href="https://github.com/google-research/bert#pre-training-with-bert" rel="nofollow noopener">explanation</a>, which is great, but I would like to use Pytorch.<br>
I’m not looking to finetune the model, just pretrain it further on the IMDB dataset, starting with an already trained model.<br>
Also, I assume that extracting BERT weights without the masked LM head and loading it into a regular BERT model should not be an issue.</p>

Jerry_Chi: <p>Did you figure out how to do this?</p>

maria: <p>For now, I’m going with the original BERT repo in tf, than will try converting to torch. as explained <a href="https://huggingface.co/transformers/converting_tensorflow_models.html" rel="nofollow noopener">here</a>.</p>

krishansubudhi: <p>Use this repo<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/microsoft/AzureML-BERT" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/6154722?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/microsoft/AzureML-BERT" target="_blank" rel="nofollow noopener">microsoft/AzureML-BERT</a></h3>

<p>End-to-end recipes for pre-training and fine-tuning BERT using Azure Machine Learning service - microsoft/AzureML-BERT</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>This is slightly old but works. Let me know if you face any issues.</p>

maria: <p>This introduces the overhead of using Azure.<br>
I was looking for an example using the pytorch-pretrained-bert or transformers, maybe something with the model zoo - all native to Pytorch and agnostic to infrastructure.<br>
Thanks though.</p>

krishansubudhi: <p>This uses huggungface Bert. If you can set the rank and world size explicitly, then you don’t need to use azure.</p>

maria: <p>Using the original Google repo and converting to torch works.<br>
[EDIT] one caveat though, the new model is much heavier and needs more computational time and power to use.</p>

macwiatrak: <p>This might not be exactly what you are looking for, but it might be easier to follow the huggingface interface for finetuning (very well documented), augment it for a given task and then just use the hidden states of the BERT model to get the embeddings.</p>

maria: <p>I specifically wanted to do the additional pretraining.<br>
It is one if the steps used in some papers in order to adjust the distribution.<br>
The masked lm and next sentence classifier are available in huggingface, but I haven’t seen any training examples.</p>

macwiatrak: <p><a href="https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py" rel="nofollow noopener">This</a> looks relevant. You can see there an example of LM task, you can reuse it/build on it and create your own LM task inside which you will initialize the weights of bert with a pretrained version and then train it with your own data.</p>

Tykat: <p>The link <a class="mention" href="/u/macwiatrak">@macwiatrak</a> provided is giving a 404 back. Could you please provide an alternative link to the same page if still available?</p>

maria: <p>I actually ended up using the official repo for additional pretraining and exporting to PyTorch.</p>

Tykat: <p>Hi!<br>
Thank you for your answer!<br>
Could you please provide the repo of your code if it is publicly available?</p>
<p>Thank you.</p>

spadel: <p><a class="mention" href="/u/maria">@maria</a> I would be interested too <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

maria: <aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/google-research/bert" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/43830688?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/google-research/bert" target="_blank" rel="noopener nofollow ugc">google-research/bert</a></h3>

<p>TensorFlow code and pre-trained models for BERT. Contribute to google-research/bert development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


spadel: <p>But that’s just the official Tensorflow version, do you also have your PyTorch version available?</p>

maria: <p>I used tensorflow and than converted that to PyTorch.<br>
The official implementation is very user friendly.<br>
That’s part of the reason I didn’t post the solution here <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:">.</p>

BramVanroy: <p>You can use the scripts provided in the <code>transformers</code> library. These are well-tested and provide many useful options. For BERT in particular, you can have a look <a href="https://github.com/huggingface/transformers/tree/master/examples/language-modeling#robertabertdistilbert-and-masked-language-modeling" rel="noopener nofollow ugc">here</a>. If you have additional questions, you can use <a href="https://discuss.huggingface.co/" rel="noopener nofollow ugc">their dedicated forum</a>.</p>

maria: <p>Awesome! I’m glad to see examples (didn’t find any at the time - 6mo ago).</p>

m-nlp-q: <p>Hello <a class="mention" href="/u/maria">@maria</a> ,</p>
<p>did you use the <a href="https://github.com/huggingface/transformers/tree/master/examples/language-modeling#robertabertdistilbert-and-masked-language-modeling" rel="noopener nofollow ugc">scripts</a> of the <code>transformers</code> library?</p>
<p>If you did, which one do you suggest, Hugging Face or the official repo of BERT?<br>
The latter includes using TensorFlow and then export to PyTorch. Ideally, I would like to use just PyTorch and no TensorFlow.</p>