Orthogonal gradient to input image

MLAlex1: <p>Hi,</p>
<p>assuming I have computed the gradient of a loss of a neural network wrt an image of say shape (3, 224, 224), how do I then compute the corresponding orthogonal gradient?</p>
<p>Thanks!</p>

KFrank: <p>Hi Alex!</p>
<aside class="quote no-group" data-username="MLAlex1" data-post="1" data-topic="202342" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mlalex1/48/59670_2.png" class="avatar"> MLAlex1:</div>
<blockquote>
<p>how do I then compute the corresponding orthogonal gradient?</p>
</blockquote>
</aside>
<p>Think of your image and its gradient as both being vectors of length <code>3 * 224 * 224</code>.</p>
<p><code>gradient - dot (gradient, image) * image / |image|**2</code> is the component<br>
of <code>gradient</code> that is orthogonal to <code>image</code> (where <code>dot()</code> indicates the “dot” or inner<br>
product between the two vectors <code>gradient</code> and <code>image</code> and <code>|image|</code> is the norm<br>
or length of the vector <code>image</code>).</p>
<p>Best.</p>
<p>K. Frank</p>

MLAlex1: <p>Hi Frank, thanks for your reply!<br>
Let me write it in psudocode to make sure I understand:</p>
<pre><code class="lang-auto">import torch

# define random image
img = torch.rand(3, 224, 224)
img = img.requires_grad_(True)

# compute the loss
loss = Net(img)
loss.backward()

# store grad
img_grad = img.grad

# view(-1) to transform into vectors of size 3 * 224 * 224
orth_vector = img_grad.view(-1) - \
              img_grad.view(-1) @ img.view(-1) * img.view(-1) / (img.view(-1).norm() ** 2)

# check they are orthogonal
assert(orth_vector @ img_grad == 0)

</code></pre>
<p>I checked and the last assertion isn’t true for something like that. What is wrong in my pseudocode?</p>
<p>I had this idea:</p>
<pre><code class="lang-auto">import torch

# define random image
img = torch.rand(1, 3, 224, 224)
img = img.requires_grad_(True)

# compute the loss
loss = Net(img)
loss.backward()

# store grad
img_grad = img.grad

orthonormal_mtx = []
# loop through each channel in the image, so we orthonormalize for 
# each channel independently
for channel in range(3):
   # here I assume we have defined a function that performs
   # gram-schmidt orthonormalization
   orth_matrix = gram_schmidt(img_grad[channel]).unsqueeze(0) # shape (1, 224, 224)

# I can then check that each channel component is orthogonal 
# to the original gradient, for example:
# img_grad[0] @ orth_matrix[0].T will give an Identity matrix
orth_matrix = torch.cat(orth_matrix)  # shape (3, 224, 224)

</code></pre>
<p>Cheers!</p>

KFrank: <p>Hi Alex!</p>
<aside class="quote no-group" data-username="MLAlex1" data-post="3" data-topic="202342" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mlalex1/48/59670_2.png" class="avatar"> MLAlex1:</div>
<blockquote>
<pre><code class="lang-auto"># check they are orthogonal
assert(orth_vector @ img_grad == 0)
</code></pre>
<p>I checked and the last assertion isn’t true for something like that.</p>
</blockquote>
</aside>
<p>If I understand correctly what you are asking, the problem is that <code>@</code> is giving you<br>
matrix multiplication, rather than a scalar dot product.</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.3.0'
&gt;&gt;&gt; imag = torch.randn (3, 224, 224)
&gt;&gt;&gt; grad = torch.randn (3, 224, 224)
&gt;&gt;&gt; orth = grad - torch.dot (imag.flatten(), grad.flatten()) * imag / imag.norm()**2
&gt;&gt;&gt; (orth * imag).sum()
tensor(0.0006)
&gt;&gt;&gt; ((orth / orth.norm()) * (imag / imag.norm())).sum()
tensor(3.2596e-09)
</code></pre>
<p>(which is orthogonality up to expected round-off error).</p>
<p>Best.</p>
<p>K. Frank</p>