Inplace operation probleme during transfer learning

username: koopa31
cooked: <p>Hi<br>
I made a Napari plugin in which I train a CNN in a qt thread_worker. When my CNN is trained from scratch I don’t get any problem, but when the network is pretrained, I always get this type of error at some point:  <em>RuntimeWarning: RuntimeError in aborted thread: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [4, 32, 1, 1]] is at version 9; expected version 8 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!!</em><br>
Each 20 epochs, I do<br>
training_thread.pause()<br>
model.eval() <span class="hashtag-raw">#prediction</span> code<br>
traing_thread.resume()<br>
It seems the error occurs when I make the prediction even if I use <em>with torch.no_grad()</em>. The weirdest thing is that it only happens when pretrained, whereas the CNN are the same only the weights differ. Any idea?</p>

username: KFrank
cooked: <p>Hi Clément!</p>
<aside class="quote no-group" data-username="koopa31" data-post="1" data-topic="185399" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/koopa31/48/52727_2.png" class="avatar"> koopa31:</div>
<blockquote>
<p><em>one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [4, 32, 1, 1]] is at version 9; expected version 8 instead.</em></p>
</blockquote>
</aside>
<p>To debug, see if you can find the tensor of the reported shape.  You can<br>
then use, for example, <code>print (t._version)</code> to locate the operation that<br>
changes its <code>._version</code> from <code>8</code> to <code>9</code>.  That will be the problematic inplace<br>
operation that you will need to fix.</p>
<aside class="quote no-group">
<blockquote>
<p><em>Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later.</em></p>
</blockquote>
</aside>
<p>It looks like you have run this with <code>with torch.autograd.detect_anomaly():</code>.<br>
What is the forward-call backtrace that <code>detect_anomaly()</code> produces telling<br>
you?  It can be very helpful for debugging if you read through it carefully.</p>
<aside class="quote no-group">
<blockquote>
<p>training_thread.pause()<br>
model.eval() <span class="hashtag-raw">#prediction</span> code<br>
traing_thread.resume()<br>
It seems the error occurs when I make the prediction even if I use <em>with torch.no_grad()</em>.</p>
</blockquote>
</aside>
<p>Just to be clear, <code>model.eval()</code> does <em>not</em> use <code>with torch.no_grad()</code>, if<br>
that’s what you were thinking.  (Instead, <code>model.eval()</code> turns off things like<br>
<code>Dropout</code>.)</p>
<p>Also, the <code>training_thread.pause()</code> stuff looks suspicious.  I could well<br>
imagine that if you pause the the thread during, say, a backward pass,<br>
perform a forward pass (for example, for evaluation), and then resume<br>
the backward pass, you could break things with an inplace modification.</p>
<p>Best.</p>
<p>K. Frank</p>

username: koopa31
cooked: <p>As I am working in a thread, it does not show the operation which causes problems. But, I noticed that each time it crashes, the responsible tensor is of different shape. Hence, it seems it is not related to a single operation, and the one which makes it all crash down is random.</p>
<p>model.eval() does not refer to model.eval() of Pytorch, model is a python object containing a net object. Hence model.eval(params) is a custom function for evaluation in which we make model.net.eval().</p>
<p>Other weird thing, it only crashes for transfer learning andf it only crashes in the second case of this funciton:</p>
<pre><code>    if dessin_widget.little_res_window_button.value:
        training_thread.pause()

        # Plot the loss values
        loss_curve = loss_plot.plot(loss_list, pen=(255, 102, 0), clear=True)

        # Refresh the plot
        loss_plot.autoRange()

        channels = [dessin_widget.chan.choices.index(dessin_widget.chan.value),
                    dessin_widget.chan2.choices.index(dessin_widget.chan2.value)]
        square_coords = dessin_widget.viewer.value.layers["little window"].data[0]
        # Récupérer les limites de la région à extraire
        x_min = max(int(np.min(square_coords[:, 0])), 0)
        x_max = int(np.max(square_coords[:, 0]))
        y_min = max(int(np.min(square_coords[:, 1])), 0)
        y_max = int(np.max(square_coords[:, 1]))

        cropped_image = dessin_widget.viewer.value.layers["image"].data[x_min:x_max, y_min:y_max].copy()

        mask_labels = model.eval(cropped_image, diameter=int(dessin_widget.diameter_field.value),
                                 flow_threshold=float(dessin_widget.flow_th_field.value),
                                 cellprob_threshold=float(dessin_widget.cell_th_field.value),
                                 channels=channels, omni=OMNI, channel_axis=2)[0]
        dessin_widget.viewer.value.layers["CP result"].data = np.zeros_like(dessin_widget.viewer.value.layers["CP result"].data)
        dessin_widget.viewer.value.layers["CP result"].data[x_min:x_max, y_min:y_max] = mask_labels
        dessin_widget.viewer.value.layers["CP result"].refresh()
        training_thread.resume()
    else:
        training_thread.pause()
        # Plot the loss values
        loss_curve = loss_plot.plot(loss_list, pen=(255, 102, 0), clear=True)

        # Refresh the plot
        loss_plot.autoRange()

        if (epoch - 1) % int(dessin_widget.show_res_each_button.value) == 0:

            channels = [dessin_widget.chan.choices.index(dessin_widget.chan.value),
                        dessin_widget.chan2.choices.index(dessin_widget.chan2.value)]
            mask_labels = model.eval(dessin_widget.viewer.value.layers["image"].data,
                                     diameter=int(dessin_widget.diameter_field.value),
                                     flow_threshold=float(dessin_widget.flow_th_field.value),
                                     cellprob_threshold=float(dessin_widget.cell_th_field.value),
                                     channels=channels, omni=OMNI, channel_axis=2)[0]

            dessin_widget.viewer.value.layers["CP result"].data = mask_labels
            dessin_widget.viewer.value.layers["CP result"].refresh()
        training_thread.resume()
</code></pre>
<p>Finally, The threadworker yields to this function at the end of an epoch one the backward() hasa been computed, hence it should not be a problem. Moreover, why would it work in all cases except in transfer learning when predicting the whole image instead of a little crop ?</p>
<p>PS: I think I noticed something important: Every time I make a prediction when training resumes, the loss function makes a jump like it suddenly increases. It seems like the inference changes some weights even if it shouldn’t, how can it be possible?</p>

username: koopa31
cooked: <p>As I don’t have a clean solution, I found a brut force solution for the moment:</p>
<pre><code class="lang-auto">from copy import deepcopy
predict_model = deepcopy(model)
mask_labels = predict_model.eval(dessin_widget.viewer.value.layers["image"].data,
                         diameter=int(dessin_widget.diameter_field.value),
                         flow_threshold=float(dessin_widget.flow_th_field.value),
                         cellprob_threshold=float(dessin_widget.cell_th_field.value),
                         channels=channels, omni=OMNI, channel_axis=2)[0]
</code></pre>
<p>I know it’s absolutely not optimal from a memory point of view, but at least it works until I find a cleaner solution.</p>

username: KFrank
cooked: <p>Hi Clément!</p>
<aside class="quote no-group" data-username="koopa31" data-post="3" data-topic="185399" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/koopa31/48/52727_2.png" class="avatar"> koopa31:</div>
<blockquote>
<p>As I am working in a thread, it does not show the operation which causes problems.</p>
</blockquote>
</aside>
<p>I’ve never looked at <code>detect_anomaly()</code>'s forward-call backtrace when it<br>
was produced inside a thread, but are you sure it isn’t providing correct<br>
information?  It would seem to me that it would still work, even in a thread.</p>
<aside class="quote no-group">
<blockquote>
<p>each time it crashes, the responsible tensor is of different shape. Hence, it seems it is not related to a single operation, and the one which makes it all crash down is random.</p>
</blockquote>
</aside>
<p>This kind of “randomness” is a very common symptom of threading issues.</p>
<p>QThreads have a well-deserved reputation for being difficult to use correctly.<br>
(I’m not up to date on the QThreads documentation, but in the past it included<br>
some technically-incorrect information and offered some bad design advice.)</p>
<aside class="quote no-group">
<blockquote>
<p>andf it only crashes in the second case of this funciton:</p>
<pre><code>    if dessin_widget.little_res_window_button.value:
        training_thread.pause()
</code></pre>
</blockquote>
</aside>
<p>You pause a thread, run your model, restart the thread, and you get an<br>
inplace-modification error the <em>second</em> time you do this.  This is almost<br>
surely the result of a threading problem.</p>
<aside class="quote no-group">
<blockquote>
<p>The threadworker yields to this function at the end of an epoch one the backward() hasa been computed, hence it should not be a problem.</p>
</blockquote>
</aside>
<p>In principle this sounds okay, but there is a lot of circumstantial evidence<br>
pointing to a threading problem.</p>
<p>Hypothetically, if <code>training_thread.pause()</code> sends an (asynchronous)<br>
pause signal to your thread_worker and then returns (without blocking),<br>
the call to <code>model.eval()</code> could muck up the computation graph before<br>
your threadworker had, in fact, safely completed its epoch.  (This is just<br>
one example of the kinds of things that could go wrong.)</p>
<aside class="quote no-group quote-modified" data-username="koopa31" data-post="4" data-topic="185399">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/koopa31/48/52727_2.png" class="avatar"> koopa31:</div>
<blockquote>
<p>I found a brut force solution for the moment:</p>
<pre><code class="lang-auto">from copy import deepcopy
predict_model = deepcopy(model)
mask_labels = predict_model.eval (...)
</code></pre>
</blockquote>
</aside>
<p>This does suggest that <code>model.eval()</code> is somehow messing up the<br>
computation graph.  The <code>deepcopy()</code> creates new tensors that are not<br>
part of the original computation graph, potentially protecting you from<br>
the inplace-modification error.  That is, <code>model.eval()</code> may still be<br>
(somehow) modifying the deepcopied model’s computation graph inplace,<br>
but this doesn’t cause an error when <code>training_thread</code> calls <code>.backward()</code><br>
and / or <code>optimizer.step()</code> on the original model.</p>
<p>I would nonetheless advise getting to the bottom of your original issue.<br>
Because of their “randomness,” threading bugs can cause infrequent,<br>
“unexpected” errors.  The concern is that your brute-force “solution”<br>
might be sweeping the real issue under the rug by eliminating one of<br>
the frequent (or more obvious) consequences of the core bug, but not<br>
eliminating less frequent (or less obvious) errors.</p>
<p>Good luck!</p>
<p>K. Frank</p>