My Model does perform, but not when created via a function

BaoTai_Le: <p>Hello everybody,<br>
i am currently trying to implement a loop for testing and training over many experiments and stumbled across a peculiar problem!</p>
<p>I am working with a Triplett Training Model ( Embedding Layer + Triplett Wrapping) and when I build the model step by step in Jupyter Notebook the training loop (<strong>triplett_train_and_test</strong>) works just fine!</p>
<blockquote>
<p><span class="hashtag-raw">#Batch_size</span>, Epochs, etc.<br>
TRAIN_BATCH_SIZE = BATCH_SIZE<br>
TEST_BATCH_SIZE = int(BATCH_SIZE*VAL_SPLIT)<br>
EPOCHS = 100<br>
NUM_HARD = NUM_HARD<br>
EMB_SIZE = 10<br>
DROPOUT_PROB = 0.1<br>
EMBEDDING_DIM = 14</p>
</blockquote>
<blockquote>
<p><span class="hashtag-raw">#Instantiate</span> the Embedding Model<br>
EMBEDDING_MODEL = Simple_Embedding_V1(embedding_dim = EMBEDDING_DIM,<br>
num_embeddings=EMB_SIZE,dropout_prob=DROPOUT_PROB )</p>
</blockquote>
<blockquote>
<p><span class="hashtag-raw">#Instantiate</span> the  Triplett Wrapper with the underlying Embedding Model<br>
MODEL = Triplett_V0(EMBEDDING_MODEL)</p>
</blockquote>
<blockquote>
<p><span class="hashtag-raw">#Learning</span> Rate, Optimizer, scheduler, loss_fn for training<br>
lr = 0.001<br>
optimizer = torch.optim.Adam(params=MODEL.parameters(), lr =lr)<br>
scheduler=torch.optim.lr_scheduler.ExponentialLR(optimizer, gamma=0.9)<br>
loss_fn = nn.TripletMarginLoss(margin =1.0,eps = 1e-7)</p>
</blockquote>
<p>However when I try to use the following function to create the model</p>
<pre><code>def create_Triplett_V0(embedding_model, input_size, output_size, dropout_prob=0):

    # 1. Get the base model with random weights and send to device

    model=embedding_model(embedding_dim = input_size, 
    num_embeddings=output_size,dropout_prob=dropout_prob)
    model.apply(init_weights)

    MODEL =Triplett_V0(model)

    #2. Make sure that the parameters are trainable
    for param in MODEL.parameters():
        param.requires_grad= True
    
    #3. Set the seeds (just sets the random seed to 42 to have more comparable results)
    set_seeds()

    #4. Give the model a name
    model_name = f"Triplett_V0_{model.name}"
    print(f"[INFO] Created new {model_name} model.")
    return MODEL
</code></pre>
<p>the test and train_loss stay stagnant.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/6/666621a09fcb1ff4dfc408e7f77fa1608766efe1.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/666621a09fcb1ff4dfc408e7f77fa1608766efe1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/666621a09fcb1ff4dfc408e7f77fa1608766efe1_2_690x474.jpeg" alt="image" data-base62-sha1="eBRw4V96gEcvaLkwkdgq44RGHzX" width="690" height="474" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/666621a09fcb1ff4dfc408e7f77fa1608766efe1_2_690x474.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/6/6/666621a09fcb1ff4dfc408e7f77fa1608766efe1.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/6/666621a09fcb1ff4dfc408e7f77fa1608766efe1.jpeg 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">920×633 90.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>For what it’s worth I have already checked and compared <strong>requires_grad</strong>, <strong>grad_fn</strong>, <strong>is_leaf</strong> of the underlying embedding layer the Triplett Model between those two approaches and could not find anything, that would be different between them.</p>
<p>If anybody could help me out, I would be very grateful! If you guys need any further information let me know!</p>
<p>Happy Coding to everyone!</p>

ptrblck: <p>It seems you are creating different models based on your code.<br>
First approach uses:</p>
<pre data-code-wrap="python"><code class="lang-python">EMBEDDING_MODEL = Simple_Embedding_V1(embedding_dim = EMBEDDING_DIM,
num_embeddings=EMB_SIZE,dropout_prob=DROPOUT_PROB )
MODEL = Triplett_V0(EMBEDDING_MODEL)
</code></pre>
<p>while the second approach uses:</p>
<pre data-code-wrap="python"><code class="lang-python">model=embedding_model(embedding_dim = input_size, 
    num_embeddings=output_size,dropout_prob=dropout_prob)
MODEL =Triplett_V0(model)
</code></pre>
<p>so the “base” model seems to differ besides the additional <code>init_weights</code> call.</p>

BaoTai_Le: <p>My mistake i forgot to mention the following cell</p>
<blockquote>
<p>triplett_simple_v1= create_Triplett_V0(Simple_Embedding_V1, input_size =14, output_size =10)</p>
</blockquote>
<p>so the embedding_model in my function call create_Triplett_V0 should also  be Simple_Embedding_V1</p>
<p>But after you mentioned that I might have used different models I went and check the summary of the two models and both look exactly like this</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/e/6e9374ababf213432f4053bbc17c91f683d3f24d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/6e9374ababf213432f4053bbc17c91f683d3f24d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e9374ababf213432f4053bbc17c91f683d3f24d_2_418x500.jpeg" alt="image" data-base62-sha1="fMcrxBfrvLHzEJ5Jtt92vBBFl7v" width="418" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e9374ababf213432f4053bbc17c91f683d3f24d_2_418x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/6/e/6e9374ababf213432f4053bbc17c91f683d3f24d.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/e/6e9374ababf213432f4053bbc17c91f683d3f24d.jpeg 2x" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">620×740 99.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
( As this is my first post i cannot show you both summaries, so you have to believe me, that both look the same)</p>