How does loss function affect model during training?

username: seberino
cooked: <p>This post was flagged by the community and is temporarily hidden.</p>
cooked_hidden: yes
hidden_reason_id: 3

username: ptrblck
cooked: <p>The criterion is independent from the model and they “communicate” through the training process. I.e. the criterion calculates the loss, which is then used for the gradient calculation. The optimizer will then update the passed parameters such that the model reduces the loss.</p>

username: seberino
cooked: <p>This post was flagged by the community and is temporarily hidden.</p>
cooked_hidden: yes
hidden_reason_id: 3

username: ptrblck
cooked: <p>Why would the model or optimizer need a handle to the criterion?<br>
The <code>loss.backward()</code> call will calculate the gradients and will assign (or accumulate) these to the <code>.grad</code> attributes of all parameters. The loss is connected to the model via the computation graph and thus the <code>backward</code> pass has access to all used parameters. There is no need to pass the criterion handle around as it doesn’t contain anything and is just providing the loss calculation.</p>

username: seberino
cooked: <p>This post was flagged by the community and is temporarily hidden.</p>
cooked_hidden: yes
hidden_reason_id: 3

username: ptrblck
cooked: <p>Yes, this is done via Autograd, which creates a computation graph and assigns valid backward functions to the <code>.grad_fn</code> attribute of activation tensors.</p>

username: seberino
cooked: <p>Thanks. Here is original code layed out…</p>
<p><a href="https://blog.paperspace.com/writing-cnns-from-scratch-in-pytorch/" rel="noopener nofollow ugc">https://blog.paperspace.com/writing-cnns-from-scratch-in-pytorch/</a></p>
<p>What line there does this “Autograd” piece and creates bindings<br>
between “criterion” and/or loss “model” ?</p>
<p>cs</p>

username: ptrblck
cooked: <p>PyTorch will track all differentiable operations on tensors requiring gradients:</p>
<pre><code class="lang-python">x = torch.randn(1, 1, requires_grad=True)

y = x * 2
print(y.grad_fn)
# &lt;MulBackward0 object at 0x7f161e967490&gt;

# loss can be anything
loss = y**2
loss.backward()

print(x.grad)
# tensor([[-8.4696]])
</code></pre>
<p><a href="https://pytorch.org/docs/stable/notes/autograd.html">This doc</a> and <a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html">this tutorial</a> might be good starters.</p>

username: seberino
cooked: <p>Sorry to press this issue. If I can humbly ask one more question…</p>
<p>Yes that makes perfect sense because your line “<em>y = x * 2</em>” and then “<em>loss = y**2</em>”<br>
associates <em>x</em> with <em>loss</em>.</p>
<p>In my aforementioned link to the code and tutorial<br>
there is nothing equivalent to that! My aforementioned link uses these 2 lines…</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()
..
loss = criterion(outputs, labels)
</code></pre>
<p>Where does the analog to your <em>x</em> get introduced!?</p>

username: ptrblck
action_code: visible.enabled

username: ptrblck
cooked: <p>The model’s <code>outputs</code> tensor is attached to a computation graph as trainable parameters were used. Instead of the <code>x</code> tensor any <code>.weight</code> and/or <code>.bias</code> of layers was used, which also creates a computation graph:</p>
<pre><code class="lang-python">x = torch.randn(1, 10)
lin = nn.Linear(10, 10)

out = lin(x)
loss = out.mean()
loss.backward()
</code></pre>

username: seberino
cooked: <p>Yes that’s another one that is readily understood.  The <em>loss</em> object is tied to <em>x</em> through the <em>out = lin(x)</em> line.</p>
<p>My example…</p>
<p>criterion = nn.CrossEntropyLoss()<br>
…<br>
loss = criterion(outputs, labels)</p>
<p>had the loss set to some generic instance of nn.CrossEntropyLoss <strong>without</strong><br>
any relation to the model or my equivalent of <em>x</em> to use your nomenclature.</p>
<p>Thanks!</p>
<p>Chris</p>

username: ptrblck
cooked: <p><code>x</code> won’t receive any gradients in my latest example and you can verify it by accessing its <code>.grad</code> attribute, which will return <code>None</code>. It’s just the input to the model and the <code>lin.weight</code> and <code>lin.bias</code> attributes are now the leaf tensors receiving gradients.<br>
The analogy is that a computatiom graph will be created by applying differentiable operations using trainable parameterrs. The model’s output corresponds to the <code>y</code> tensor and the <code>lin.weight</code> and <code>lin.bias</code> correspond to <code>x</code> from my first example.<br>
The loss function is just applying other differentiable operations (in the same way a linear layer is performing a matmul).</p>

username: seberino
cooked: <p>Bottom line is all of your examples make sense.<br>
The only example I can’t understand is the one<br>
with</p>
<p>criterion = nn.CrossEntropyLoss()<br>
…<br>
loss = criterion(outputs, labels)</p>
<p>It this case the loss has no connection to any tensor anywhere?!</p>
<p>cs</p>

username: ptrblck
cooked: <p>It does since <code>output</code> was created by the model as mentioned a few times already and as seen in the linked blog post:</p>
<pre><code class="lang-python">        # Forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)
</code></pre>

username: seberino
cooked: <p>Thanks.  I think I get it now.  This seems like an oddity in the way PyTorch does things…</p>
<p>So you’re saying in “<em>outputs = model(images)</em>”, that the <em>outputs</em> object will have more<br>
than just the numerical predictions!?  <em>outputs</em> will <strong>also</strong> have a link/handle (whatever you want to call it)<br>
to the <em>model</em> object?  I can accept that but that seems weird.  Did I get it right now?</p>
<p>cs</p>

username: ptrblck
cooked: <p>Yes, your explanation is correct. The “link” from the <code>outputs</code> tensor to the model’s parameters is done by Autograd and reflected in the computation graph and the <code>outputs.grad_fn</code> object. A very naive point of view would be to think about Autograd “recording” the operations in the forward pass by creating the computation graph. In the backward pass the <code>grad_fn</code> will be used to backpropagate through the entire graph. Internals might be more complicated (e.g. PyTorch is smart enough to figure out when to stop the backpropagation if no gradients are needed in previous operations).</p>

username: Christopher_Mills
cooked: <p>This is an excellent walkthrough of a nuanced aspect of the PyTorch computation graph. I had the exact same question, and this discussion helped me.</p>