Multiple network with one loss function, non-differentiable, is it okay with backpropagation?

chan_so: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0c02eafb834066a145884fc1f0d7f48b925015ba.png" data-download-href="https://discuss.pytorch.org/uploads/default/0c02eafb834066a145884fc1f0d7f48b925015ba" title="non-diff"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c02eafb834066a145884fc1f0d7f48b925015ba_2_690x233.png" alt="non-diff" data-base62-sha1="1IfYPNHs7hHCrCCDBCdT7FoiqpQ" width="690" height="233" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c02eafb834066a145884fc1f0d7f48b925015ba_2_690x233.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c02eafb834066a145884fc1f0d7f48b925015ba_2_1035x349.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c02eafb834066a145884fc1f0d7f48b925015ba_2_1380x466.png 2x" data-dominant-color="727B6A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">non-diff</span><span class="informations">1851×627 40.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Hello I’m trying to make the augmentation function which chooses to augmentate or not based on the contents of image.</p>
<p>To make this, I’m thinking about using two different network.</p>
<p>One is to classify augmentating the image or not The ohter is to classify the image based on label (Main task)</p>
<p>Network A makes output with False or True (0,1) to decide input image should be augmentated or not. Based on result of Network A, the image is augmentated and fed in to Network B to classify it with comparing to ground truth label.</p>
<p>I’m wondering that is this model will correctly backpropagate and learns correctly to increase the perfomance of image classification.</p>
<p>Because Network A and B is seprated image augmentation functtion which is non-differentiable.</p>
<p>Sorry for my poor explanation, but I want to know is this method will theoretically work.</p>

KFrank: <p>Hi Chan!</p>
<aside class="quote no-group" data-username="chan_so" data-post="1" data-topic="174723" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chan_so/48/58296_2.png" class="avatar"> chan_so:</div>
<blockquote>
<p>Because Network A and B is seprated image augmentation functtion which is non-differentiable.</p>
</blockquote>
</aside>
<p>You are correct that because the information that Network A passes to<br>
Network B is a discrete variable (zero or one), that connection will not<br>
be usefully differentiable, and you will not be able to use backpropagation<br>
and gradient descent to train Network A.</p>
<p>Here are two possible approaches:</p>
<p>Plan A:  Have Network A pass a continuous value (say, any value between<br>
zero and one, or perhaps from <code>-inf</code> to <code>inf</code>) to Network B, and have Network<br>
B not augment the image if the value is zero, fully augment the image if the<br>
value is one, and “partially” augment the image if the value is in between.  Of<br>
course such a scheme only makes sense if your use case has a sensible<br>
notion of “partial augmentation.”</p>
<p>Plan B:  Have Network B always classify both the unaugmented image and<br>
the augmented image, compute the loss function for both images separately,<br>
and combine the two loss-function values together using the continuous<br>
value passed to Network B by Network A (say, by taking a weighted average<br>
of the two loss values).  Then backpropagate that combined loss value through<br>
Network A (and use the resulting gradients to train Network A).</p>
<p>Of course, whether or not such schemes will work in practice will depend on<br>
the details of your actual problem, but they illustrate plausible ways to make the<br>
connection between Networks A and B continuous and usefully differentiable.</p>
<p>Best.</p>
<p>K. Frank</p>