How to train a neural network model with bert embeddings instead of static embeddings like glove/fasttext?

username: arjunsankarlal
cooked: <p>I am looking for some heads up to train a conventional neural network model with bert embeddings that are generated dynamically (BERT contextualized embeddings which generates different embeddings for the same word which when comes under different context).</p>
<p>In normal neural network model, we would initialize the model with glove or fasttext embeddings like,</p>
<pre><code class="lang-auto">import torch.nn as nn 

embed = nn.Embedding(vocab_size, vector_size)

embed.weight.data.copy_(some_variable_containing_vectors)
</code></pre>
<p>Instead of copying static vectors like this and use it for training, I want to pass every input to a BERT model and generate embedding for the words on the fly, and feed them to the model for training.</p>
<p>So should I work on changing the forward function in the model for incorporating those embeddings?</p>
<p>Any help would be appreciated!</p>

username: Justin_Sybrandt
cooked: <p>Take a look at the HuggingFace “transformers” repo.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/huggingface/transformers">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/huggingface/transformers" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/295;"><img src="https://repository-images.githubusercontent.com/155220641/a16c4880-a501-11ea-9e8f-646cf611702e" class="thumbnail" width="690" height="295"></div>

<h3><a href="https://github.com/huggingface/transformers" target="_blank" rel="noopener nofollow ugc">GitHub - huggingface/transformers: 🤗 Transformers: State-of-the-art Machine...</a></h3>

  <p>🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX. - GitHub - huggingface/transformers: 🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, a...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>This can download the pretrained Bert embeddings of your choice, and gives you a pretty straightforward interface for tokenization and extracting embeddings.</p>
<p>The rough outline of your code will look like this:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/3/63c6457d212b4582f19e46f368bef2f072471cff.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/63c6457d212b4582f19e46f368bef2f072471cff" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/63c6457d212b4582f19e46f368bef2f072471cff_2_690x360.jpeg" alt="image" data-base62-sha1="eeE3YH7kY503dmK870guQWE4Rxd" width="690" height="360" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/63c6457d212b4582f19e46f368bef2f072471cff_2_690x360.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/63c6457d212b4582f19e46f368bef2f072471cff_2_1035x540.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/63c6457d212b4582f19e46f368bef2f072471cff_2_1380x720.jpeg 2x" data-dominant-color="F3EDDB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1690×883 199 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>