How to create a 'multiply layer' which supports backprop?

Jarvis: <p>I am trying to find a function which could help me to element-wise product two feature map, and supports backprop at the meantime, like the layer in keras( <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f50128419b8acfd07cc5abf0b86cde17eb63c304.png" data-download-href="https://discuss.pytorch.org/uploads/default/f50128419b8acfd07cc5abf0b86cde17eb63c304" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f50128419b8acfd07cc5abf0b86cde17eb63c304_2_690x214.png" alt="image" data-base62-sha1="yXpsuVCxLokEA5lu2YQoYD4Zlhq" width="690" height="214" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f50128419b8acfd07cc5abf0b86cde17eb63c304_2_690x214.png, https://discuss.pytorch.org/uploads/default/original/3X/f/5/f50128419b8acfd07cc5abf0b86cde17eb63c304.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/5/f50128419b8acfd07cc5abf0b86cde17eb63c304.png 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1002×312 15.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Nikronic: <p>Hi,</p>
<p>PyTorch uses a semantic called <code>autograd</code> to handle backward operations automatically. So the only thing you need to take care of is the <code>forward</code> pass of your custom layer.</p>
<p>First you define a class that extends <code>torch.nn.Module</code>:</p>
<pre><code class="lang-python">class Multiply(nn.Module):
  def __init__(self):
    super(Multiply, self).__init__()

  def forward(self, tensors):
    result = torch.ones(tensors[0].size())
    for t in tensors:
      result *= t
    return t
</code></pre>
<p>And your input must support <code>grad</code>, you can use below snippet:</p>
<pre><code class="lang-python">x = torch.randn(3, 2, 2)
x.requires_grad = True  # this makes sure the input `x` will support grad ops
</code></pre>
<p>Now, whenever you want, you can call <code>backward</code> on any tensors that passed through this layer or the output of this layer itself to calculate grads for you.</p>
<p>The below code just uses the output to calculate grad of the multiplied layers:</p>
<pre><code class="lang-python">multiply = Multiply()
y = multiply([x, x])
torch.sum(y).backward()
</code></pre>
<p>And the reason I used <code>torch.sum</code> is that backward only is possible on scalar outputs.<br>
Here is the <a href="http://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-custom-nn-modules" rel="nofollow noopener">official tutorial of creating custom layers</a>, it will help for sure.</p>
<p>Bests,<br>
Nik</p>

mfoglio: <p>If I use <code>torch.multiply</code> will the backpropagation be handled correctly? I am currently using it to multiply two variables (output of convolutions) inside the <code>forward</code> method of a custom module (extends <code>nn.Module</code>).</p>