Model not re-producible with pretrained embeddings and freeze=False

username: ironv
cooked: <p><strong>NN arch</strong></p>
<pre><code class="lang-auto">[input] -&gt; [embeddings] -&gt; [BiLSTM] (two hidden layers torch.cat) -&gt; 
    [FC layers incl dropout] -&gt; [output]
</code></pre>
<p><strong>Code Structure</strong></p>
<ul>
<li>grid search for best learning rate, batch size, FC - <span class="hashtag">#layers</span>, width, etc.</li>
<li>chose best model based on F1 on validation</li>
<li>re-train final model from scratch using “best” parameters from grid search, and compute metrics</li>
</ul>
<p>Seeds have been set before EVERY training run <code>torch.manual_seed(some value)</code> and     <code>torch.cuda.manual_seed_all(some value)</code></p>
<p>The following works fine (best from grid search get <em>perfectly</em> replicated)</p>
<ol>
<li>training entire network starting with uninitialized embeddings (object instantiated as  <code>nn.Embedding(#embed, edim, padding_idx=0)</code>)</li>
<li>importing embeddings and setting up embedding layer using<br>
<code>nn.Embedding.from_pretrained(my_embeddings, freeze=True, padding_idx=0)</code>
</li>
</ol>
<p>When the freeze flag is changed to False, the model is <strong>not</strong> reproducible!  The code is EXACTLY the same as before (as in 2).  The only thing changed is the <code>freeze=False</code>.</p>
<p>Should I be setting some other seed? Or is this a known issue?</p>

username: ironv
cooked: <p>FWIW, here’s how I could work around the above issue based on information posted in this group.<br>
[<a href="https://discuss.pytorch.org/t/word2vec-as-input-to-lstm/40356">link1</a>] [<a href="https://discuss.pytorch.org/t/partially-freeze-embedding-layer/18458/6">link2</a>]. I implemented three variants of using pre-trained embeddings and the models are <em>reproducible</em>.</p>
<ol>
<li>read pre-trained embeddings and FREEZE</li>
</ol>
<pre><code class="lang-auto">embed = nn.Embedding(num_embeddings, embedding_dim, padding_idx=0)

#embed_init is a numpy array with the embeddings
embedpt = torch.from_numpy(embed_init).float().to(device)
#ind_init is a numpy array with indices of words for which embedding are available
indpt = torch.from_numpy(ind_init).long().to(device)

#after model object has been instantiated
assert model.embed.weight.shape == embedpt.shape
model.embed.weight.data.copy_(embedpt)
model.embed.weight.requires_grad = False
</code></pre>
<ol start="2">
<li>use PT embeddings to initialize (may be better than initializing with random values)</li>
</ol>
<p>(same as 1) except <code>model.embed.weight.requires_grad = True</code></p>
<ol start="3">
<li>freeze embeddings if available, train embeddings which are not available eg. vocab has 1000 words of which you have embeddings for 700 of them which you would like to freeze, but train the other 300.</li>
</ol>
<p>(same as 2) additionally in the training loop</p>
<pre><code class="lang-auto">optimizer.zero_grad()
loss.backward()
model.embed.weight.grad[indpt] = 0  &lt;=======
optimizer.step()
</code></pre>
<p>My code is set up to do an extended grid search on multiple gpus including a <code>patience</code> parameter (<span class="hashtag">#epochs</span> with no improvement in monitored quality after which training will be stopped).  The best parameter set including number of epochs is saved. When the final model is run <code>patience</code> is turned off.  The final model now re-traces the steps of the selected grid search run, which was not happening when with the function calls in the first post.</p>