Two issues I'm encountering while optimizing an Ellipse to a Circle as a beginner

username: MartensCedric
cooked: <p>I have pretty simple code</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import random

image_width, image_height = 128, 128

def apply_ellipse_mask(img, pos, axes):
    mask = torch.zeros_like(img)
    for r in range(image_height):
        for c in range(image_width):
            val = ((c - pos[0])**2) / axes[0]**2 + ((r - pos[1])**2) / axes[1]**2
            assert not torch.isnan(val)
            mask[r][c] = torch.where(0.9 &lt; val &lt; 1, torch.tensor(1.0),  torch.tensor(0.0))

    return img * (1.0 - mask) + mask

random.seed(0xced)

sphere_radius = image_height / 3
sphere_position = torch.tensor([image_width / 2, image_height / 2 ,0], requires_grad=True)

ref_image = apply_ellipse_mask(torch.zeros(image_width, image_height, requires_grad=True), sphere_position, [sphere_radius, sphere_radius, sphere_radius])

ellipsoid_pos = torch.tensor([sphere_position[0], sphere_position[1], 0], requires_grad=True)
ellipsoid_axes = torch.tensor([image_width / 3 + (random.random() - 0.5) * image_width / 5, image_height / 3 + (random.random() - 0.5) * image_height / 5, image_height / 2], requires_grad=True)

optimizer = torch.optim.Adam([ellipsoid_axes], lr=0.1)
criterion = torch.nn.MSELoss()
for _ in range(100):

    optimizer.zero_grad()
    current_image = torch.zeros(image_width, image_height, requires_grad=True)
    current_image = apply_ellipse_mask(current_image, ellipsoid_pos, ellipsoid_axes)

    # mixed_img = (((ref_image + current_image) / 2.0) * 255).byte()
    # mixed_img = Image.fromarray(mixed_img.numpy(), mode='L')
    # mixed_img.show()
    loss = criterion(current_image, ref_image)
    loss.backward()
    print(_, loss)
    optimizer.step()

</code></pre>
<p>However, I’m encoutering two issues that as a Pytorch beginner I find hard to understand.</p>
<ol>
<li>The first one is that this code throws an error.</li>
</ol>
<blockquote>
<p>RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.</p>
</blockquote>
<p>I don’t understand <em>why</em> it would be trying to backward through the same graph a second time? or am I directly accessing saved tensors after they were freed? I don’t understand why this happens.</p>
<ol start="2">
<li>The other issue is the following:</li>
</ol>
<p><code>ellipsoid_axes.grad</code> is None for my function. I believe the issue arises in</p>
<p><code>apply_ellipse_mask</code> in the line<br>
<code>mask[r][c] = torch.where(0.9 &lt; val &lt; 1, torch.tensor(1.0),  torch.tensor(0.0))</code>. How can I fix this?</p>

username: KFrank
cooked: <p>Hi Cedric!</p>
<aside class="quote no-group" data-username="MartensCedric" data-post="1" data-topic="209971" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/martenscedric/48/72949_2.png" class="avatar"> MartensCedric:</div>
<blockquote>
<p>I have pretty simple code</p>
</blockquote>
</aside>
<p>You have several problems here, but let me start with a core<br>
conceptual issue:</p>
<p>In order to compute your ellipse-mismatch loss function, you<br>
compare two zero-one mask images.  The problem is that<br>
such a loss function isn’t (usefully) differentiable.  As you vary,<br>
for example, <code>ellipsoid_axes</code>, some pixel will remain at zero<br>
over some range and then discontinuously change to one,<br>
where it will remain over some further range.  The derivative<br>
of that pixel with respect to <code>ellipsoid_axes</code> will be zero over<br>
the ranges where the pixel is constant and undefined (or <code>inf</code>,<br>
if you prefer) where the pixel jumps from zero to one.</p>
<p>Pytorch uses gradient descent to minimize loss functions, but<br>
when your gradient is zero (almost) everywhere, gradient<br>
descent doesn’t have the information it needs to know in which<br>
direction in parameter space to move to make the loss function<br>
smaller.</p>
<p>(If you were to use some other algorithm that could minimize<br>
your non-differential loss function effectively, you would be<br>
able to fit your ellipse reasonably well.)</p>
<p>You could make your current general scheme work by letting<br>
your “masked” pixel values vary continuously from zero to<br>
one as you cross your <code>0.9 &lt; val &lt; 1</code> boundaries.  Then your<br>
<code>MSELoss</code> loss function would end up being differentiable, and<br>
gradient descent would work to optimize it.</p>
<aside class="quote no-group">
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">ref_image = apply_ellipse_mask(torch.zeros(image_width, image_height, requires_grad=True), sphere_position, [sphere_radius, sphere_radius, sphere_radius])
</code></pre>
</blockquote>
</aside>
<p>You define several tensors incorrectly by initializing them with<br>
<code>requires_grad=True</code>.  <code>ref_image</code> is one of them.</p>
<p>As a general rule, you only want to set <code>requires_grad=True</code><br>
for tensors that you will be training (optimizing) and that<br>
therefore need to have gradients with respect to them<br>
computed.</p>
<p>In your case these would be <code>ellipsoid_pos</code> and <code>ellipsoid_axes</code>.</p>
<aside class="quote no-group">
<blockquote>
<ol>
<li>The first one is that this code throws an error.</li>
</ol>
<blockquote>
<p>RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed).</p>
</blockquote>
</blockquote>
</aside>
<p>This error is sort of moot, but it’s worth understanding why it<br>
occurs in your specific (incorrect) code.</p>
<p>You run:</p>
<pre data-code-wrap="python"><code class="lang-python">ref_image = apply_ellipse_mask(torch.zeros(image_width, image_height, requires_grad=True), sphere_position, [sphere_radius, sphere_radius, sphere_radius])
</code></pre>
<p>once, outside of your optimization loop.  The (unnamed) tensor<br>
to which you apply <code>apply_ellipse_mask()</code> is defined with<br>
<code>requires_grad = True</code> and therefore when you call<br>
<code>loss.backward()</code>, autograd backpropagates through<br>
the call to <code>apply_ellipse_mask()</code>, releasing the part of the<br>
computation graph that connects <code>ref_image</code> to the unnamed<br>
tensor.  Because <code>ref_image</code> is computed only once, outside<br>
of the optimization loop, that part of the graph is not rebuilt<br>
and you get your backward-a-second-time error when<br>
<code>loss.backward()</code> is called a second time in the second<br>
iteration of your optimization loop.</p>
<aside class="quote no-group">
<blockquote>
<p><code>ellipsoid_axes.grad</code> is None for my function. I believe the issue arises in</p>
<p><code>apply_ellipse_mask</code> in the line<br>
<code>mask[r][c] = torch.where(0.9 &lt; val &lt; 1, torch.tensor(1.0),  torch.tensor(0.0))</code>.</p>
</blockquote>
</aside>
<p>This is correct.  <code>val</code> is connected to <code>ellipsoid_axes</code> in<br>
pytorch’s computation graph, but, because your <code>torch.where()</code><br>
is not (usefully) differentiable, <code>mask[r][c]</code> is not connected<br>
to <code>val</code>.  So backpropagation never reaches <code>ellipsoid_axes</code><br>
and its gradient is not set.</p>
<aside class="quote no-group">
<blockquote>
<p>How can I fix this?</p>
</blockquote>
</aside>
<p>As outlined above, make your loss function (and all steps<br>
that lead up to it) (usefully) differentiable and <code>loss.backward()</code><br>
will backpropagate to <code>ellipsoid_axes</code> and populate its<br>
gradient.</p>
<p>Best.</p>
<p>K. Frank</p>