Multiple cost optimization on output layer and bottleneck layer of a Conv AE

Tomojit: <p>Hello - I’m working on a Convolutional Autoencoder model where I’m optimizing one cost on output layer and other cost on the bottleneck layer. Please see the picture  below(I took it from a published paper). Please note I’m not minimizing any clustering loss on the bottleneck layer. My cost is a different one. I know how to back propagate the reconstruction loss but I’m not sure how to back propagate the gradient of the cost which I’m optimizing on the output of the bottleneck layer. Can anyone please help? Thank you.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/1/21b59c2b7c9eeeaa2e4f5f1c926766592ec2a02f.png" data-download-href="https://discuss.pytorch.org/uploads/default/21b59c2b7c9eeeaa2e4f5f1c926766592ec2a02f" title="DCECLoss"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/21b59c2b7c9eeeaa2e4f5f1c926766592ec2a02f_2_690x268.png" alt="DCECLoss" data-base62-sha1="4OcQXGOBMKOVL2KJkMIiwYpFbOv" width="690" height="268" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/21b59c2b7c9eeeaa2e4f5f1c926766592ec2a02f_2_690x268.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/21b59c2b7c9eeeaa2e4f5f1c926766592ec2a02f_2_1035x402.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/21b59c2b7c9eeeaa2e4f5f1c926766592ec2a02f_2_1380x536.png 2x" data-dominant-color="F2F5F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">DCECLoss</span><span class="informations">2187×850 94.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Could you explain a bit how you are calculating the “gradient of the cost” and how the gradients should be calculated?<br>
Based on the description it seems you would like to calculate the 2nd derivative?</p>

Tomojit: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> -  My cost functions are standard MSE loss. I’m not calculating 2nd derivative. This is what I’m doing.</p>
<ol>
<li>From the forward pass capture the output of the bottleneck layer and the output layer.</li>
</ol>
<h1>Forward pass</h1>
<p>output_en,output_de = self.forward(trInput)</p>
<ol start="2">
<li>
<p>Then calculate the reconstruction loss using MSE from the activation from the output layer.<br>
<span class="hashtag">#calculate</span> reconstruction loss<br>
loss = criterion(output_de,trOutput)</p>
</li>
<li>
<p>Now calculate the cost from the output of the bottleneck layer.<br>
secondCost = self.secondCost(output_en,L)</p>
</li>
<li>
<p>Now add the second cost with the loss.<br>
loss = loss + secondCost</p>
</li>
<li>
<p>Now call the backward pass to update parameters.</p>
</li>
</ol>
<h1>Backward and optimize</h1>
<p>optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()</p>
<p>Do you think this is the correct way? I’m unsure whether the parameters of the encoder will be updated or not by the second cost.</p>

Tomojit: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> To be more specific: For the second loss I just want to update the parameters of the encoder, and I want to update the parameters of the whole network (encoder+decoder) for the reconstruction loss. But I’m not sure whether or not my current implementation will achieve what I want.</p>
<p>Thank you.</p>

ptrblck: <p>Your implementation is correct and the parameters of the encoder would accumulate the gradients from both losses, while the parameters of the decoder would only get the gradients from the reconstruction loss.</p>

Tomojit: <p>Thank you for confirming this!</p>

Tomojit: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> - I have one more query. I want to optimize a different cost using the activation from the bottleneck layer. I will calculate the cost using log of some quantity of bottleneck layer. This is what I’m thinking…</p>
<ol>
<li>
<p>From the forward pass capture the output of the bottleneck layer and the output layer.<br>
output_en,output_de = self.forward(trInput)</p>
</li>
<li>
<p>First I’m calculating the reconstruction loss as shown below.<br>
loss = criterion(output_de,trOutput)</p>
</li>
<li>
<p>Then calculate the new cost using the bottleneck output.<br>
M := some quantity I’m calculating from output_en<br>
loss2 = -log(M)</p>
</li>
<li>
<p>Now add the second cost with the loss.<br>
loss = loss + loss2</p>
</li>
<li>
<p>Backward and optimize<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()</p>
</li>
</ol>
<p>The reconstruction loss is the standard MSE so I know that Torch can calculate the derivative. But the 2nd loss is a custom one. I’m not sure whether Torch can calculate the derivative. Do I need to write a new criterion for the 2nd cost? If so then how to do that?</p>
<p>Thank you,<br>
Tomojit</p>

ptrblck: <p>As long as you are using PyTorch operations Autograd would be able to create the computation graph and calculate the gradients of the loss w.r.t. to all parameters in this graph.<br>
You can verify it by checking the <code>.grad</code> attributes of all encoder parameters before the first <code>backward</code> call (should be <code>None</code>) and afterwards (should have valid values).</p>