Simple Linear/Sigmoid Network not learning

SamyGarib: <p>I’m learning pytorch and tried to train a network as an XOR gate. Everything runs smoothly, but it just does not learn. It does changes it weights, yet it converges in a result for every input that is way out of the expected results.</p>
<p>I have tried with many learning rates and weights initialization.</p>
<p>So the inputs are A and B gates and it should return 1 if both are equals or 0 otherwise, like this :</p>
<pre>    [0,0] =&gt; 1
    [0,1] =&gt; 0
    [1,0] =&gt; 0
    [1,1] =&gt; 1

</pre>
<p>This is my attempt of modeling and training the model:</p>
<pre>    import torch as torch
    import torch.nn as nn
    
    class Network(nn.Module):
        
        def __init__(self):
            super(Network, self).__init__()
            self.x1 = nn.Linear(2,4)
            self.s1 = nn.Sigmoid()
            self.x2 = nn.Linear(4,1)
            self.s2 = nn.Sigmoid()
        
        def init(self):
            nn.init.uniform_(self.x1.weight)
            nn.init.uniform_(self.x2.weight)
    
        def forward(self, feats):
            f1 = torch.tensor(feats).float()
            xr1= self.x1(f1)
            xs1= self.s1(xr1)
            xr2= self.x2(xs1)
            out= self.s2(xr2)        
            return out  
    
        def train(self,val_expected,feats_next):
            val_expected_tensor = torch.tensor(val_expected)
            criterion = nn.MSELoss()
            optimizer = torch.optim.SGD(self.parameters(), lr=0.01)
            def closure():
                optimizer.zero_grad()
                resp = self.forward(feats_next)
                error = criterion(resp,val_expected_tensor)
                error.backward()
                return error
            optimizer.step(closure)
    
    net = Network()
    net.init()
    
    for input in ([0.,0.],[0.,1.],[1.,0.],[1.,1.]):
        response=net.forward(input)
        print(response)
    
    print ("--TRAIN START-")
    for i in range(1000):
        net.train([1.],[0.,0.])
        net.train([0.],[1.,0.])
        net.train([0.],[0.,1.])
        net.train([1.],[1.,1.])
    print ("---TRAIN END---")
    
    for input in ([0.,0.],[0.,1.],[1.,0.],[1.,1.]):
        response=net.forward(input)
        print(response)

</pre>
<p>This is a run with 100000 iterations at 0.001 learning rate:</p>
<pre>    tensor([0.7726], grad_fn=)
    tensor([0.7954], grad_fn=)
    tensor([0.8229], grad_fn=)
    tensor([0.8410], grad_fn=)
    --TRAIN START-
    *.........*.........*.........*.........*.........*.........*.........*.........*.........*.........
    ---TRAIN END---
    tensor([0.6311], grad_fn=)
    tensor([0.6459], grad_fn=)
    tensor([0.6770], grad_fn=)
    tensor([0.6906], grad_fn=)

</pre>
<p>I’m really lost here. Shound’t this work?</p>

KFrank: <p>Hello Samy!</p>
<aside class="quote no-group" data-username="SamyGarib" data-post="1" data-topic="98714" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/samygarib/48/29580_2.png" class="avatar"> SamyGarib:</div>
<blockquote>
<p>train a network as an XOR gate.</p>
</blockquote>
</aside>
<p>I think it is rather difficult to train a small network to reproduce an XOR<br>
gate.</p>
<p>XOR has cross-terms that are, in some sense, highly non-linear.  This<br>
is an imperfect fit for a “narrow,” “shallow” network.</p>
<p>Please take a look at this thread about a nearly identical problem:</p>
<aside class="quote quote-modified" data-post="1" data-topic="90913">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pemfir/48/43803_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/simple-pattern-is-not-detected-by-simple-linear-model/90913">Simple pattern is not detected by simple linear model</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    I have a very simple example of 6 training data with 6 labels. I am trying to fit a simple logistic regression to it, and hoping it should over-fit with 0 loss, but the model does not converge. 
Here is the rule, i am hoping the model to learn: 

When input is  [1., 0., 0., 1.] , the label is 1
When input is  [0., 1., 1., 0.] , the label is 1
When input is  [1., 0., 1., 0.] , the label is 0
When input is  [0., 1., 0., 1.] , the label is 0

is such pattern something a non-linear model cannot capt…
  </blockquote>
</aside>

<p>Some further comments in line:</p>
<aside class="quote no-group">
<blockquote>
<p>I have tried with many learning rates and weights initialization.</p>
</blockquote>
</aside>
<p>Continue to experiment with learning rates and many random<br>
initializations, and continue to train for many iterations.  It looks<br>
like it is easy for the training of this kind of network to get stuck<br>
away from a good solution.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">        def __init__(self):
            super(Network, self).__init__()
            self.x1 = nn.Linear(2,4)
            self.s1 = nn.Sigmoid()
            self.x2 = nn.Linear(4,1)
</code></pre>
</blockquote>
</aside>
<p>Play around with making your network <em>deeper</em> by adding one or<br>
more hidden layers, e.g., something like:</p>
<pre data-code-wrap="python"><code class="lang-python">            self.x1 = nn.Linear(2,4)
            self.x2 = nn.Linear(4,2)
            self.x3 = nn.Linear(2,1)
</code></pre>
<p>or:</p>
<pre data-code-wrap="python"><code class="lang-python">            self.x1 = nn.Linear(2,4)
            self.x2 = nn.Linear(4,4)
            self.x3 = nn.Linear(4,2)
            self.x4 = nn.Linear(2,1)
</code></pre>
<p>Try making your network <em>wider:</em></p>
<pre data-code-wrap="python"><code class="lang-python">            self.x1 = nn.Linear(2,16)
            self.x2 = nn.Linear(16,1)
</code></pre>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">            self.s2 = nn.Sigmoid()
</code></pre>
</blockquote>
</aside>
<p>I would suggest getting rid of the <code>Sigmoid</code>.  My intuition is that it will<br>
actually hurt here – you have to saturate the <code>Sigmoid</code> (large network<br>
parameters) to match closely your <code>val_expected</code>.  Return the result<br>
of your last <code>Linear</code> layer as the output of your network.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">            criterion = nn.MSELoss()
</code></pre>
</blockquote>
</aside>
<p>Conventional wisdom suggests that <code>BCEWithLogitsLoss</code> would be<br>
a more natural loss function for your problem.  You should be able<br>
to get your network to train with <code>MSELoss</code>, but maybe not as easily.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">            optimizer = torch.optim.SGD(self.parameters(), lr=0.01)
</code></pre>
</blockquote>
</aside>
<p>Consider using <code>momentum</code> with <code>SGD</code> (as well as various learning<br>
rates).  You might also play around with other optimizers, such as<br>
<code>Adam</code>.</p>
<p>I would first remove the final <code>Sigmoid</code>, use <code>BCEWithLogitsLoss</code>,<br>
and make the network deeper and/or wider.</p>
<p>Good luck.</p>
<p>K. Frank</p>