Adding bias to convolution output

username: Ajinkya_Ambatwar
cooked: <p>Hi,<br>
This maybe a very simple question but it’s troubling me.<br>
I am trying to write convolution using matrix multiplication using fold and unfold methods.</p>
<p>My matrix multiplication output after reshaping is (1, 24, 26, 26). I have to add bias to it which is a 1-D tensor or shape (24).<br>
I am assuming that the bias tensor has to be expanded to the shape (1,24,26,26) such that each of the 26*26 elements for each of 24 indices are same as the original value.<br>
Can someone suggest me the fastest way to add the bias value to the cnn output using pytorch functions?</p>
<p>Thanks!</p>

username: googlebot
cooked: <p>for example</p>
<blockquote>
<p>bias = nn.Parameter(torch.zeros(24,1,1))<br>
…<br>
x += bias</p>
</blockquote>
<p>inplace op += is fine for autograd</p>

username: Ajinkya_Ambatwar
cooked: <p>HI <a class="mention" href="/u/googlebot">@googlebot</a> thanks for the reply.<br>
I already have bias value stored in the given tensor. I can’t initialize it again the way you said.<br>
IN my case, I have the bias value coming from the state_dict of a pretrained network. I am trying to route it to matrix multiplication domain only for inference purpose.</p>
<p>Any comments on that?</p>

username: googlebot
cooked: <p>then just reshape it to (24,1,1) instead, this aligns dimensions so that it broadcasts with other summand</p>

username: Ajinkya_Ambatwar
cooked: <p>Hi <a class="mention" href="/u/googlebot">@googlebot</a>, I am creating this sample test to illustrate what I want.</p>
<pre><code class="lang-auto"># this is like the output of the matmul say
a = torch.tensor([[[[1,2,3], [4,5,6], [7,8,9]],[[10, 11,12], [13,14,15],[7, 8,16]]]]) 

a.shape
#prints torch.size([1,2,3,3])

#this is like bias say
b = torch.tensor([4,5])
print(b.shape)
#prints torch.size([2])

b.reshape(1,2,1,1)
#reshaping as you said

#adding
a+b
</code></pre>
<p>What I expect is</p>
<pre><code class="lang-auto">output = tensor([[[[ 1+4,  2+4,  3+4],
          [ 4+4,  5+4,  6+4],
          [ 7+4,  8+4,  9+4]],

         [[10+5, 11+5, 12+5],
          [13+5, 14+5, 15+5],
          [ 7+5,  8+5, 16+5]]]])
</code></pre>
<p>This is the way bias should be added to matmul output right?</p>
<p>What I get is</p>
<pre><code class="lang-auto">RuntimeError                              Traceback 
----&gt; 1 a+b

RuntimeError: The size of tensor a (3) must match the size of tensor b (2) at non-singleton dimension 3
</code></pre>
<p>Please  help me with this</p>

username: dunefox
cooked: <pre><code class="lang-auto">In [4]: b = b.reshape(1,2,1,1)                                                                

In [5]: a + b                                                                                 
Out[5]: 
tensor([[[[ 5,  6,  7],
          [ 8,  9, 10],
          [11, 12, 13]],

         [[15, 16, 17],
          [18, 19, 20],
          [12, 13, 21]]]])
</code></pre>
<p>Does <code>b.reshape</code> return a copy? Seems like it’s not an in-place operation.</p>

username: googlebot
cooked: <aside class="quote no-group" data-username="dunefox" data-post="6" data-topic="82684">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dunefox/48/24578_2.png" class="avatar"> dunefox:</div>
<blockquote>
<p>Does <code>b.reshape</code> return a copy?</p>
</blockquote>
</aside>
<p>It is a new tensor, but it refers to the same memory (when possible). So, yes, it acts like a copy.</p>

username: Ajinkya_Ambatwar
cooked: <p><a class="mention" href="/u/dunefox">@dunefox</a>   your method worked. Apparently just reshaping b is not inplace operation. So yes assigning the value back to b works in this case.<br>
Thanks for the help <a class="mention" href="/u/googlebot">@googlebot</a> <a class="mention" href="/u/dunefox">@dunefox</a>!!</p>