Different learnings but with updating equally the weights

ateixeira: <p>Hi, i have this code about a differential and a standard learning and they are generating different values for losses, but the weights are updating exactly the same, generating, thus, the same predictions… Id like to know if in this method is there any opportunity:</p>
<pre><code>def train_fit(self,
              early: bool,
              n_epochs: int,
              batches_per_epoch: int = 16,
              min_batch_size: int = 256,
              lr_schedule: tuple = ((0.0, 1.0e-3), (0.2, 0.1), (0.6, 0.01), (0.9, 1.0e-4), (1.0, 1.0e-6)),
              best_loss: float = float('inf'),
              patience: int = 50,
              min_delta: int = 0):
    # Set up fit method
    loss_values = []
    batch_size = max(min_batch_size, self.x.size(0) // batches_per_epoch)
    lr_schedule_epochs, lr_schedule_rates = zip(*lr_schedule)
    counter = 0
    # Run epochs
    for epoch in range(n_epochs):
        learning_rate = np.interp(epoch / n_epochs, lr_schedule_epochs, lr_schedule_rates)
        batch_losses = []
        # Run data batches
        for batch_start in range(0, self.x.size(0), batch_size):
            batch_end = min(batch_start + batch_size, self.x.size(0))
            input = self.x[batch_start:batch_end].clone().detach().requires_grad_(True if self.differential else False)
            label = self.y[batch_start:batch_end].clone().detach()
            # Training itself
            if self.differential:
                dydx_batch = self.dydx[batch_start:batch_end].clone().detach()
                output = self.forward(input)
                output.mean().backward(retain_graph=True)
                diff_output = input.grad
                loss = 0.5 * self.criterion(output, label) + 0.5 * self.criterion(diff_output * self.lambda_j, dydx_batch * self.lambda_j)
            else:
                output = self.forward(input)
                loss = self.criterion(output, label)
            batch_losses.append(loss.item())
            # Reset weight grads wrt inputs and finishes the training calculating grads wrt loss
            self.optimizer.zero_grad()
            loss.backward()
            self.optimizer.step()
        # Store epoch losses
        loss_values.append(np.mean(batch_losses))
    return loss_values
</code></pre>
<p>I know the problem can be elsewhere, but Id like to check this method…</p>

KFrank: <p>Hi Andrei!</p>
<aside class="quote no-group" data-username="ateixeira" data-post="1" data-topic="205565" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/8edcca/48.png" class="avatar"> ateixeira:</div>
<blockquote>
<pre><code>            if self.differential:
                dydx_batch = self.dydx[batch_start:batch_end].clone().detach()
</code></pre>
</blockquote>
</aside>
<p>Because of the <code>.clone().detach()</code>, <code>dydx_batch</code> won’t carry<br>
<code>requires_grad = True</code>.</p>
<aside class="quote no-group">
<blockquote>
<pre><code>                output = self.forward(input)
                output.mean().backward(retain_graph=True)
                diff_output = input.grad
                loss = 0.5 * self.criterion(output, label) + 0.5 * self.criterion(diff_output * self.lambda_j, dydx_batch * self.lambda_j)
</code></pre>
</blockquote>
</aside>
<p><code>input.grad</code> also doesn’t  carry <code>requires_grad = True</code>, so neither term in<br>
your “differential” loss:</p>
<pre data-code-wrap="python"><code class="lang-python">self.criterion(diff_output * self.lambda_j, dydx_batch * self.lambda_j)
</code></pre>
<p>will contribute gradients to your overall <code>loss</code>.</p>
<p>I suggest that you try:</p>
<pre data-code-wrap="python"><code class="lang-python">diff_output = torch.autograd.grad (output, input, create_graph = True)
</code></pre>
<aside class="quote no-group">
<blockquote>
<pre><code>            else:
                output = self.forward(input)
                loss = self.criterion(output, label)
</code></pre>
</blockquote>
</aside>
<p>One point does confuse me though:  In your <code>differential</code> branch,<br>
<code>self.criterion (output, label)</code> is multiplied by <code>0.5</code>, but not in<br>
your “regular” branch.  So I would have expected the two versions to<br>
train differently (and in the specific case of using plain-vanilla <code>SGD</code> as<br>
your optimizer, with the <code>differential</code> branch training with effectively<br>
half the learning rate).</p>
<p>Best.</p>
<p>K. Frank</p>