Is my nn.Sequential starting warm during each gridsearch loop?

username: rharwood
cooked: <p>I recently migrated my project from sk-learn to pytorch, so my implementation might be way off.</p>
<p>I am successively training an nn with different hyperparameter values to perform a grid search. During this process I am seeing the R^2 score gradually increase for each combination of hyperparameter values. This is not expected. So I am wondering if the nn’s weights are not actually being cleared between iterations. Below is my implementation:</p>
<pre><code class="lang-auto">regr = nn.Sequential(
    nn.Linear(6, 20),
    nn.ReLU(),
    nn.Linear(20,20),
    nn.ReLU(),
    nn.Linear(20,20),
    nn.ReLU(),
    nn.Linear(20,20),
    nn.ReLU(),
    nn.Linear(20,1)
).to(device)
</code></pre>
<pre><code class="lang-auto">for hyperparameter in hyperparameters:
    optimizer = torch.optim.Adam(regr.parameters(), lr=0.001)
    for epoch in range(n_epochs):
        loss = custom_loss()
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
</code></pre>
<p>I would have expected each new hyperparameter loop to create a new instance of the optimizer object. This may not be the case. What is a better approach?</p>
<p>I have left out some details. I am hoping this is enough but I am happy to provide more code if not.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="rharwood" data-post="1" data-topic="215139">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/e19adc/48.png" class="avatar"> rharwood:</div>
<blockquote>
<p>I would have expected each new hyperparameter loop to create a new instance of the optimizer object. This may not be the case. What is a better approach?</p>
</blockquote>
</aside>
<p>You are creating a new optimizer, but you are not creating a new model or resetting its parameters.<br>
<code>loss = custom_loss()</code> also seems to perform the forward pass and loss computation, so unsure where the model is created.</p>

username: rharwood
cooked: <p>That’s true. Here is my CustomLoss class:</p>
<pre><code class="lang-auto">class CustomLoss(nn.Module):
    def __init__(self, weight_1=1.0, weight_2=1.0):
        super(CustomLoss, self).__init__()
        # Weights to control the influence of each term
        self.weight_1 = weight_1
        self.weight_2 = weight_2
        # Define the loss functions
        self.mse_loss = nn.MSELoss()

    def forward(self, predictions, true_values, theoretical_values):
        # Loss Term 1: Error between predictions and true values (ground truth)
        loss_term_1 = self.mse_loss(predictions, true_values)
        
        # Loss Term 2: Error between predictions and theoretical values
        loss_term_2 = self.mse_loss(predictions, theoretical_values)
        
        # Combine the two loss terms
        total_loss = self.weight_1 * loss_term_1 + self.weight_2 * loss_term_2
        return total_loss
</code></pre>
<p>How should I go about creating a new model or resetting the parameters? Thank you!</p>

username: ptrblck
cooked: <p>You could call the <code>reset_parameters()</code> method on all registered modules. Something like this should work:</p>
<pre data-code-wrap="python"><code class="lang-python">for module in model.modules():
    if hasattr(module, "reset_parameters"):
        module.reset_parameters()
</code></pre>
<p>or you could simply recreate the model and optimizer:</p>
<pre data-code-wrap="python"><code class="lang-python">regr = nn.Sequential(
    nn.Linear(6, 20),
    nn.ReLU(),
    nn.Linear(20,20),
    nn.ReLU(),
    nn.Linear(20,20),
    nn.ReLU(),
    nn.Linear(20,20),
    nn.ReLU(),
    nn.Linear(20,1)
).to(device)
optimizer = torch.optim.Adam(regr.parameters(), lr=0.001)
</code></pre>
<p>inside the <code>hyperparamters</code> loop.</p>

username: rharwood
cooked: <p>The model is created here.</p>
<aside class="quote no-group" data-username="rharwood" data-post="1" data-topic="215139">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/e19adc/48.png" class="avatar"> rharwood:</div>
<blockquote>
<pre><code class="lang-auto">regr = nn.Sequential(
    nn.Linear(6, 20),
    nn.ReLU(),
    nn.Linear(20,20),
    nn.ReLU(),
    nn.Linear(20,20),
    nn.ReLU(),
    nn.Linear(20,20),
    nn.ReLU(),
    nn.Linear(20,1)
).to(device)
</code></pre>
</blockquote>
</aside>
<p>It seems like my main issue was creating my model outside of the hyperparameter loop.</p>
<p>Yes, this should fix my issues. Thank you, <a class="mention" href="/u/ptrblck">@ptrblck</a> !</p>