Classification neural Network using Autograd

username: msee19018
cooked: <p>Hey guys, I want to make a simple classification neural network with pytorch’s autograd package. I have gone through some resources that helped me create the code. Problem is the code is not working I have tried some solutions but it does not work for me.<br>
I am trying to classify mnist dataset, I am building simple 4 layer network using matrix multiplications. I am using sigmoid as activation and cross entropy loss function. Network processes code in batches. My request to guys is please have look at code and tell me what i am doing wrong in code? Code the network class and training loop in given below. Link to colab file also given. Thanks in advance <a href="https://colab.research.google.com/drive/1V4mfk4Fbc7x4uCIehmSohEVEXwJAnlOn?" rel="noopener nofollow ugc">https://colab.research.google.com/drive/1V4mfk4Fbc7x4uCIehmSohEVEXwJAnlOn?</a>usp=sharing</p>
<p>class network():</p>
<pre><code>def __init__(self,num_layer,layer_size ): 

    self.W1=torch.randn(layer_size[0],layer_size[1],dtype=torch.float32,requires_grad=True)

    self.b1=torch.randn(layer_size[1],dtype=torch.float32,requires_grad=True)

    self.W2=torch.randn(layer_size[1],layer_size[2],dtype=torch.float32,requires_grad=True)

    self.b2=torch.randn(layer_size[2],dtype=torch.float32,requires_grad=True)

    

    self.W3=torch.randn(layer_size[2],layer_size[3],dtype=torch.float32,requires_grad=True)

    self.b3=torch.randn(layer_size[3],dtype=torch.float32,requires_grad=True)

    self.W4=torch.randn(layer_size[3],layer_size[4],dtype=torch.float32,requires_grad=True)

    self.b4=torch.randn(layer_size[4],dtype=torch.float32,requires_grad=True)

    self.act=torch.nn.ReLU()

    self.act1=torch.nn.Sigmoid()

    self.act_last=torch.nn.Softmax(dim=1)

    self.loss1 = torch.nn.CrossEntropyLoss()

def forward(self,x):

    return self.act1(torch.matmul(self.act1(torch.matmul(self.act1(torch.matmul(self.act1(torch.matmul(x,self.W1)+self.b1),self.W2) +self.b2),self.W3) +self.b3),self.W4)+self.b4)



def loss(self,pred,Y):

    return -1*(torch.sum(Y*(torch.log(pred))))/20

    # return l
</code></pre>
<p>model=network(5,[784,392,196,98,10])</p>
<p>lr=0.00001</p>
<p>n_iters=100</p>
<p>for epoch in range(n_iters):</p>
<pre><code>for i in range (3000):

    value = randint(0, train_X.shape[0]-50)

    

    X1=X[value:value+20]

    Y1=Y[value:value+20]

    y_pred=model.forward(X1)

    # print(Y1)

    # print(y_pred)

    l=model.loss(y_pred,Y1)

    

    l.backward()

    with torch.no_grad():

        model.W1-=lr*model.W1.grad

        model.W2-=lr*model.W2.grad

        model.W3-=lr*model.W3.grad

    # value_W4=self.W4-lr*self.W4.grad

        model.b1-=lr*model.b1.grad

        model.b2-=lr*model.b2.grad

        model.b3-=lr*model.b3.grad

    model.W1.grad.zero_()

    model.W2.grad.zero_()

    model.W3.grad.zero_()

    model.b1.grad.zero_()

    model.b2.grad.zero_()

    model.b3.grad.zero_()

print(f'epoch {epoch+1},loss={l:.8f}')</code></pre>

username: KFrank
cooked: <p>Hi Hamad!</p>
<p>First, I think going through an exercise like this is very worthwhile.<br>
Taking the time to do “by hand” some of the things built in to pytorch<br>
is a great way to really learn what is going on and will prove valuable<br>
when you move on to tackling more complicated problems.</p>
<aside class="quote no-group" data-username="msee19018" data-post="1" data-topic="116724" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/msee19018/48/31260_2.png" class="avatar"> msee19018:</div>
<blockquote>
<p>I want to make a simple classification neural network with pytorch’s autograd package.</p>
</blockquote>
</aside>
<p>I have not gone through your code in detail, so there may be other<br>
issues, but I do see a couple of things that you should look into.</p>
<aside class="quote no-group">
<blockquote>
<p>class network():</p>
<pre><code>def __init__(self,num_layer,layer_size ): 
</code></pre>
<p>…</p>
<pre><code>    self.act=torch.nn.ReLU()
</code></pre>
</blockquote>
</aside>
<p>As a minor note, you might want to remove or comment out things<br>
such as <code>self.act</code> that are unused.  Doing so could make your code<br>
more readable, and perhaps less error-prone.</p>
<aside class="quote no-group">
<blockquote>
<pre><code>    self.act1=torch.nn.Sigmoid()
</code></pre>
<p>…</p>
<pre><code>def forward(self,x):

    return self.act1(torch.matmul(self.act1(torch.matmul(self.act1(torch.matmul(self.act1(torch.matmul(x,self.W1)+self.b1),self.W2) +self.b2),self.W3) +self.b3),self.W4)+self.b4)
</code></pre>
</blockquote>
</aside>
<p>Applying a <code>Sigmoid</code> layer to your final linear layer is not correct for<br>
a multi-class classification problem where you use cross-entropy as<br>
your loss.  Doing so will convert each of your ten outputs into an<br>
<em>independent</em> probability (between <code>0.0</code> and <code>1.0</code>), but you want a<br>
probability distribution over the ten classes.  That is, you want a set<br>
of ten values that are not only individually between <code>0.0</code> and <code>1.0</code>,<br>
but that sum to <code>1.0</code> as well.</p>
<p>To get such a probability distribution you want your final activation<br>
layer to be your <code>self.act_last=torch.nn.Softmax(dim=1)</code> .</p>
<p>(However, what you <em>really</em> want is <em>not</em> to have a final activation layer,<br>
to work directly with the raw-score <em>logits</em> generated by your final linear,<br>
and to write a “cross-entropy-with-logits” loss function that uses<br>
<a href="https://pytorch.org/docs/1.8.1/nn.functional.html#torch.nn.functional.log_softmax" rel="noopener nofollow ugc">log_softmax()</a> internally.)</p>
<aside class="quote no-group">
<blockquote>
<pre><code>def loss(self,pred,Y):

    return -1*(torch.sum(Y*(torch.log(pred))))/20
</code></pre>
</blockquote>
</aside>
<p>You don’t say what <code>Y</code> is here, but, as written, <code>Y</code> should be <em>one-hot</em><br>
encoded class labels (or more generally, “soft labels” that are given<br>
by a probability distribution over the classes – each in <code>[0.0, 1.0]</code>,<br>
and that sum to <code>1.0</code>).</p>
<aside class="quote no-group">
<blockquote>
<p>model=network(5,[784,392,196,98,10])</p>
<p>lr=0.00001</p>
<p>n_iters=100</p>
</blockquote>
</aside>
<p>Some side comments:</p>
<p>As it currently stands, your network has what we call three “hidden”<br>
layers.  I would also encourage you to experiment with “shallower”<br>
networks with only one or two hidden layers.</p>
<p>Your learning rate of <code>0.00001</code> strikes me as being smaller than it<br>
needs to be.  You might be able to train faster with a larger learning<br>
rate (although it is sometimes necessary to start with a smaller rate<br>
during a “warm-up” period and it can be helpful to reduce the rate<br>
towards the end of your training to avoid jumping back and forth<br>
across “gullies.”)</p>
<p>You will almost certainly need more that <code>100</code> epochs.</p>
<aside class="quote no-group">
<blockquote>
<pre><code>    with torch.no_grad():

        model.W1-=lr*model.W1.grad

        model.W2-=lr*model.W2.grad

        model.W3-=lr*model.W3.grad

    # value_W4=self.W4-lr*self.W4.grad
</code></pre>
</blockquote>
</aside>
<p>Now, the big one:  You aren’t optimizing your final linear layer.  This<br>
will make it very hard for your network to train, as the upstream<br>
layers will have to “learn” to undo the damage being done by your<br>
randomly-initialized – and frozen – final layer.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: msee19018
cooked: <p>Thanks <a class="mention" href="/u/kfrank">@KFrank</a> your suggestion helped a lot. i solved my problem. Not using Softmax was perhaps the key drawback in my code.</p>