Implementing the adjoint operator of a convolutional layer

username: kof123
cooked: <p>Hello everybody,</p>
<p>that’s my first post, so please be kind!</p>
<p>I am currently trying to implement the adjoint operation of a convolutional layer (in 2D and 3D).<br>
More precisely, let c be a convoltutional layer with a pre-defined kernel, e.g. a 3x3 kernel.<br>
What is the easiest and most efficient way of implementing the adjoint operation of c, i.e. c*?<br>
Note that I am NOT looking for a transposed convolution, but I really need the adoint operator of c, i.e. it has to fulfill<br>
&lt;cx,y&gt; = &lt;x,c*y&gt; for any x and y.</p>
<p>Thank you very much in advance for any help!<br>
Cheers,<br>
Andreas</p>

username: ptrblck
cooked: <p>I’m not sure, if you mean the Hermitian adjoint, but if so I assume you would like to inverse the convolution?<br>
If so, then note that inverse filtering is often not a good idea, if it doesn’t account for the additional error signal in the input:</p>
<pre><code class="lang-auto">(f * g) + e = h
</code></pre>
<p>If you know the type of noise (e.g. Gaussian) you could use a Wiener deconvolution.</p>
<p>For a general use case you could use the FFT and reverse the (noise free) convolution via:</p>
<pre><code class="lang-auto">F = H / G
</code></pre>
<p>and apply the inverse FFT again.</p>

username: kof123
cooked: <p>exactly, i need the hermitian adjoint of the convolution-operation c, say c^H The reason is that I am considering a functional F(x) = 1/2*||cx -y||_2^2, where c is just a simple conv.layer with no activation function. Considering the derivative of F with respect to x I get c^H(cx -y). Thus, I need the adjoint operator of c. Since a conv. can in general be associated with a linear operator (i.e. a matrix), if I had the matrix as it was, it would be easy, cause I would just use A^H. But I would like to be able to express the operation A^H by means of the same function which I am using in the implementation, i.e conv2d or conv3d. I guess using FFT in that case does not help, since in the end I end up with c^H(cx -y) anyway.<br>
But thanks again <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: KFrank
cooked: <p>Hello Andreas!</p>
<aside class="quote no-group" data-username="kof123" data-post="1" data-topic="100010" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/a6a055/48.png" class="avatar"> kof123:</div>
<blockquote>
<p>I really need the adoint operator of c, i.e. it has to fulfill<br>
&lt;cx,y&gt; = &lt;x,c*y&gt; for any x and y.</p>
</blockquote>
</aside>
<p><a href="https://pytorch.org/docs/1.6.0/generated/torch.nn.ConvTranspose2d.html#torch.nn.ConvTranspose2d" rel="noopener nofollow ugc">ConvTranspose2d</a> should do what you want (at least in the plain-vanilla,<br>
2d case – I didn’t check 3d or various stride / padding combinations).</p>
<p>Here is an example script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.__version__
torch.random.manual_seed (2020)

cnva = torch.nn.Conv2d (1, 1, 3, padding = 1, bias = False)
cnvt = torch.nn.ConvTranspose2d (1, 1, 3, padding = 1, bias = False)
with torch.no_grad():
    _ = cnvt.weight.copy_ (cnva.weight)

s = torch.randn ((1, 1, 7, 7))
t = torch.randn ((1, 1, 7, 7))
(cnva (s) * t).sum()
(s * cnvt (t)).sum()
</code></pre>
<p>Note the use of the <em>same</em> weight matrix for <code>ConvTranspose2d</code> as for<br>
<code>Conv2d</code>.</p>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.6.0'
&gt;&gt;&gt; torch.random.manual_seed (2020)
&lt;torch._C.Generator object at 0x7efc1f81c6f0&gt;
&gt;&gt;&gt;
&gt;&gt;&gt; cnva = torch.nn.Conv2d (1, 1, 3, padding = 1, bias = False)
&gt;&gt;&gt; cnvt = torch.nn.ConvTranspose2d (1, 1, 3, padding = 1, bias = False)
&gt;&gt;&gt; with torch.no_grad():
...     _ = cnvt.weight.copy_ (cnva.weight)
...
&gt;&gt;&gt; s = torch.randn ((1, 1, 7, 7))
&gt;&gt;&gt; t = torch.randn ((1, 1, 7, 7))
&gt;&gt;&gt; (cnva (s) * t).sum()
tensor(3.5478, grad_fn=&lt;SumBackward0&gt;)
&gt;&gt;&gt; (s * cnvt (t)).sum()
tensor(3.5478, grad_fn=&lt;SumBackward0&gt;)
</code></pre>
<p>To understand the adjoint, think of a 2d 100x100 image as being a<br>
vector of length 1,000,000.  The 2d convolution can be represented<br>
(inefficiently) as a 1,000,000 x 1,000,000 matrix with a particular<br>
banded structure.  Because it’s real, its adjoint is just its transpose.</p>
<p><code>ConvTranspose2d</code> represents (algorithmically) the appropriate<br>
banded structure for that transposed 1,000,000 x 1,000,000 matrix.</p>
<p>(If pytorch were to support convolutions with complex weights, you<br>
would presumably use the complex conjugate of <code>Conv2d</code>’s weights<br>
for <code>ConvTranspose2d</code>.)</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: kof123
cooked: <p>Hello,<br>
perfect, this looks like what I was looking for. Exactly, I am aware of the interpretation of the conv as an application of a matrix to a vector. I knew that for 2D, one typically only has to rotate the filter to obtain the tranposed operation. But since for 3D there are multiple possibilities to rotate the filter, I was wondering if there was a function which does this without me having to take care of how exactly to manipulate the filter depending on the dimensionality of the problem.<br>
Thanks a lot again! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: KFrank
cooked: <p>Hello Andreas!</p>
<aside class="quote no-group" data-username="kof123" data-post="5" data-topic="100010" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/a6a055/48.png" class="avatar"> kof123:</div>
<blockquote>
<p>But since for 3D there are multiple possibilities to rotate the filter, I was wondering if there was a function which does this</p>
</blockquote>
</aside>
<p>Did you try this scheme with <code>Conv3d</code>?  (I didn’t.)  Did it work?</p>
<p>Best.</p>
<p>K. Frank</p>

username: kof123
cooked: <p>Hi Frank,<br>
yes, it worked fo 3D as well. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Thanks again!</p>
<p>Best,</p>
<p>Andreas</p>