PyTorch Forward Propogation

username: PURAV_SHAH
cooked: <p>Hi,<br>
This maybe a naive question to ask but i am a beginner in pytorch and i am unable to figure out how pytorch is doing the forward propagation.<br>
The code i am using is shown below-</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.cl1 = nn.Linear(5, 4)
        self.cl2 = nn.Linear(4, 2)
        
    def forward(self, x):
        print(model.cl1.weight,x)
        print(torch.mm(x,model.cl1.weight.T))
        x = self.cl1(x)
        print(x)
        x = F.relu(self.cl2(x))
        return x




model = MyModel()
x = torch.ones(1, 5)
output = model(x)
</code></pre>
<p>And this is the output i get for the above code.<br>
As you can see that after the forward propagation step the values of x calculated using matrix multiplication and the values calculated using pytorch are different.<br>
Why is this happening?am i missing something here?</p>
<pre><code class="lang-auto">Parameter containing:
tensor([[-0.0975, -0.3880, -0.2666, -0.1913,  0.3015],
        [ 0.0493, -0.3044, -0.3731,  0.2693, -0.3543],
        [ 0.0821, -0.4167, -0.2888,  0.3144,  0.3574],
        [ 0.3467,  0.4166, -0.0122, -0.0539, -0.3886]], requires_grad=True) tensor([[1., 1., 1., 1., 1.]])
tensor([[-0.6419, -0.7133,  0.0484,  0.3086]], grad_fn=&lt;MmBackward&gt;)
tensor([[-0.7729, -1.0955,  0.3968,  0.2085]], grad_fn=&lt;AddmmBackward&gt;)
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>You are missing the bias in your manual computation <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:">  You can pass <code>bias=False</code> when you create your Linear layer if you don’t want the baises.</p>

username: PURAV_SHAH
cooked: <p>Hi,<br>
Thanks for clearing that up.<br>
I have another question if you could please help me–<br>
i am trying to apply a kmeans quantization technique to the weights of each layer during the training procedure.<br>
What i mean is during the forward propagation at each layer i want to first use the kmeans algorithm to calculate the weights and then use these calculated weights and discard the old ones.<br>
Similarly the same procedure for the backpropagation step also.<br>
I am trying to use hooks to implement this functionality and for some trial purposes just to get an idea of how a hook works i have written the below code-</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.cl1 = nn.Linear(5, 4)
        self.cl2 = nn.Linear(4, 2)
        
    def forward(self, x):
        print(model.cl1.weight,model.cl1.bias)
        x = self.cl1(x)
        print(x)
        x = F.relu(self.cl2(x))
        return x


activation = {}
def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
        print(model.weight)
        model.weight.data=torch.ones(4,5)
        print(model.weight)
        output=torch.mm(input[0],model.weight.T)
        return output
    return hook

model = MyModel()
model.cl1.register_forward_hook(get_activation('fc2'))
x = torch.ones(1, 5)
output = model(x)
</code></pre>
<p>and the output i am getting is as follows–</p>
<pre><code class="lang-auto">Parameter containing:
tensor([[-0.3525,  0.4128,  0.3498,  0.0078, -0.3536],
        [-0.2505, -0.0612, -0.1943, -0.2778, -0.3931],
        [-0.2942,  0.0659,  0.0260,  0.0492, -0.2993],
        [ 0.1888,  0.2567,  0.0124,  0.3343, -0.3156]], requires_grad=True) Parameter containing:
tensor([-0.3972, -0.0496, -0.0852, -0.1048], requires_grad=True)
Parameter containing:
tensor([[-0.3525,  0.4128,  0.3498,  0.0078, -0.3536],
        [-0.2505, -0.0612, -0.1943, -0.2778, -0.3931],
        [-0.2942,  0.0659,  0.0260,  0.0492, -0.2993],
        [ 0.1888,  0.2567,  0.0124,  0.3343, -0.3156]], requires_grad=True)
Parameter containing:
tensor([[1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.]], requires_grad=True)
tensor([[5., 5., 5., 5.]], grad_fn=&lt;MmBackward&gt;)
</code></pre>
<p>As you can see from the above output i am getting the value but the grad_fn of the variable x changes.<br>
What is the best way for me to overwrite the weights of each layer while not affecting the computation graph?<br>
Thanks!</p>

username: albanD
cooked: <p>Given that you want these operations to be part of the computational graph, you should not use <code>.data</code> (in general you should never use it <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> )</p>
<p>Also I am a bit confused why you modify cl1’s weights after you did the forward on it?</p>

username: PURAV_SHAH
cooked: <p>Okay but when i try to directly modify the weights without using .data it gives me an error.</p>
<p>Oh yes i see what you mean, if you could please tell me how do we overwrite the weights before the forward propagation step?</p>

username: albanD
cooked: <p>To do it before the forward I would do the following:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.cl1 = nn.Linear(5, 4)
        self.cl2 = nn.Linear(4, 2)

        # Move the original weights so that we can change it during the forward
        # but still have the original ones detected by .parameters() and the optimizer
        self.cl1.weight_orig = self.cl1.weight
        del self.cl1.weight
        
    def forward(self, x):
        # Recompute the weight attribute based on the original value
        self.cl1.weight = get_new_weight(self.cl1.weight_orig)

        # Do the regular forward
        x = self.cl1(x)
        x = F.relu(self.cl2(x))
        return x
</code></pre>

username: PURAV_SHAH
cooked: <p>Hi,<br>
Thanks for your response!<br>
Your code works and does not affect the computation graph but i don’t think that it is taking the bias into consideration after modifying the weights.<br>
I also tried this procedure during the back propagation and don’t think i am getting the right results-</p>
<pre><code class="lang-auto">def modify_weight(weight):
    with torch.no_grad():
        if weight is not None:
            for i in range(weight.shape[0]):
                for j in range(weight.shape[1]):
                    weight[i][j]=1
    #print(weight)
    return weight
def modify_weight_grad(weight):
    if weight is not None:
        for i in range(weight.shape[0]):
            for j in range(weight.shape[1]):
                weight[i][j]=1
    #print(weight)
    return weight
class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.cl1 = nn.Linear(5, 4)
        self.cl2 = nn.Linear(4, 1)
        self.cl1.weight_orig=self.cl1.weight
        del self.cl1.weight
    def forward(self, x):
        #print(model.cl1.weight,model.cl1.bias)
        #print(torch.mm(x,model.cl1.weight.T))
        self.cl1.weight=modify_weight(self.cl1.weight_orig)
        print(self.cl1.bias)
        x = self.cl1(x)
        print(x)
        x = F.relu(self.cl2(x))
        return x



model = MyModel()
optimizer=torch.optim.SGD(model.parameters(),lr=0.01)
model.cl1.register_forward_hook(get_activation('fc2'))
x = torch.ones(1, 5)
output = model(x)
output.backward()
model.cl1.weight_orig.grad=model.cl1.weight.grad
print(model.cl1.weight.grad)
model.cl1.weight.grad=modify_weight_grad(model.cl1.weight_orig.grad)
print(model.cl1.weight.grad)
print(model.cl1.weight)
optimizer.step()
print(model.cl1.weight)
</code></pre>
<p>And the output i get is–</p>
<pre><code class="lang-auto">bias Parameter containing:
tensor([-0.2568,  0.2500,  0.1713, -0.3188], requires_grad=True)
x after forward propogation with new weight tensor([[5., 5., 5., 5.]], grad_fn=&lt;MmBackward&gt;)
Gradient of the cl1 layer tensor([[0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.]])
Gradient of cl1 layer after modification tensor([[1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.]])
weight of cl1 layer Parameter containing:
tensor([[1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.]], requires_grad=True)
weight of cl1 layer after taking an optimisation step Parameter containing:
tensor([[0.9900, 0.9900, 0.9900, 0.9900, 0.9900],
        [0.9900, 0.9900, 0.9900, 0.9900, 0.9900],
        [0.9900, 0.9900, 0.9900, 0.9900, 0.9900],
        [0.9900, 0.9900, 0.9900, 0.9900, 0.9900]], requires_grad=True)
</code></pre>
<p>As you can see from the above output the bias is not being considered and the value of the weight of cl1 layer after taking a step should be zero if i am not wrong.</p>

username: albanD
cooked: <p>Hi,</p>
<ul>
<li>You should apply the same logic to <code>.bias</code> if you want it to work as well.</li>
<li>You should NOT modify the original parameters inplace. That is why you have the weight_orig that are the original weights that are optimized. and the weight that are the ones that are the processed version of weight_orig after your kmeans like op and that is used in the forward.</li>
<li>Keep in mind that you’re doing differentiable operations. And that if you override a value, the gradient for the original value before the override is 0.</li>
</ul>

username: PURAV_SHAH
cooked: <p>Hi,<br>
Thanks for your response.</p>
<ul>
<li>
<p>I dont think i was clear about the bias question, what i was saying is that the bias is not considered at all once the weight has been modified(the forward propagation step is not adding the bias once i use the modified weights, it is simple giving me input*weight)</p>
</li>
<li>
<p>Correct me if i am wrong but am i not modifying the gradient value after it has been calculated so would it matter?i want to work with the modified gradient values.</p>
</li>
</ul>

username: albanD
cooked: <blockquote>
<p>the forward propagation step is not adding the bias once i use the modified weights</p>
</blockquote>
<p>Did you set it to zero somewhere else? Or you passed bias=False?</p>
<blockquote>
<p>Correct me if i am wrong but am i not modifying the gradient value after it has been calculated so would it matter?</p>
</blockquote>
<p>You get the gradient for what your function computes. That’s all I’m saying. If that;s what you want, then everything is fine <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: PURAV_SHAH
cooked: <p>Hi,</p>
<p>No i am not setting the bias to zero or passing bias=False anywhere.<br>
if i try and print the bias it prints it correctly but it is not using the bias during the forward propagation.</p>