XOR Model is very bad - what am I doing wrong?

AbsolutePytorchNoob: <p>I’m training a simple XOR neural network to familiarize myself with PyTorch.<br>
My code is:</p>
<pre data-code-wrap="py"><code class="lang-py">class XOR(nn.Module):
	def __init__(self):
		super().__init__()
		self.fc1 = nn.Linear(2,2)
		self.fc2 = nn.Linear(2,1)
		self.optimizer = torch.optim.Adam(self.parameters())
		self.loss = nn.BCELoss()
		self.lh = []
	def forward(self, x):
		x = torch.tanh(self.fc1(x))
		x = torch.sigmoid(self.fc2(x))
		return x
	def step(self, batch, truth):
		self.optimizer.zero_grad()
		preds = self(batch)
		loss = self.loss(preds, truth)
		loss.backward()
		self.lh.append(loss)
		self.optimizer.step()
	def fit(self, epoch):
		d1 = torch.FloatTensor([[1,1]]); t1 = torch.FloatTensor([[0]])
		d2 = torch.FloatTensor([[1,0]]); t2 = torch.FloatTensor([[1]])
		d3 = torch.FloatTensor([[0,1]]); t3 = torch.FloatTensor([[1]])
		d4 = torch.FloatTensor([[0,0]]); t4 = torch.FloatTensor([[0]])
		for _ in range(epoch):
			self.step(d1,t1)
			self.step(d2,t2)
			self.step(d3,t3)
			self.step(d4,t4)
	def test(self):
		net = self
		d1 = torch.FloatTensor([[1,1]]); d2 = torch.FloatTensor([[1,0]]); d3 = torch.FloatTensor([[0,1]]); d4 = torch.FloatTensor([[0,0]]);
		print(f" 1,1 : {net(d1).item()}\n 1,0 : {net(d2).item()}\n 0,1 : {net(d3).item()}\n 0,0 : {net(d4).item()}\n ")
	def show(self):
		l = len(self.lh)
		y = np.array(self.lh)
		x = np.arange(l)
		plt.plot(x,y)
		plt.show()
</code></pre>
<p>When I run it, this neural network is not capable of overfitting to the training data. The loss fluctuates around but no improvement is being made even when I run the fit method for 1000s of epochs. What am I doing wrong here? I am really confused. Any help would be appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Loss graph:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/6/565c9e8873b95fea07f85da484e7843ff4d8cdc2.png" data-download-href="https://discuss.pytorch.org/uploads/default/565c9e8873b95fea07f85da484e7843ff4d8cdc2" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/565c9e8873b95fea07f85da484e7843ff4d8cdc2_2_672x500.png" alt="image" data-base62-sha1="cjZuTzE1kYKqpQT1vmq4FF2r5Ci" width="672" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/565c9e8873b95fea07f85da484e7843ff4d8cdc2_2_672x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/565c9e8873b95fea07f85da484e7843ff4d8cdc2_2_1008x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/6/565c9e8873b95fea07f85da484e7843ff4d8cdc2.png 2x" data-dominant-color="BDD6E7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1157×860 130 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

KFrank: <p>Hi Dr. Noob!</p>
<aside class="quote no-group quote-modified" data-username="AbsolutePytorchNoob" data-post="1" data-topic="113175" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/848f3c/48.png" class="avatar"> AbsolutePytorchNoob:</div>
<blockquote>
<p>I’m training a simple XOR neural network</p>
<pre data-code-wrap="py"><code class="lang-py">		self.fc1 = nn.Linear(2,2)
		self.fc2 = nn.Linear(2,1)
</code></pre>
</blockquote>
</aside>
<p>You have a “hidden layer” with two “neurons.”  I believe that this<br>
does not have enough structure to capture XOR (and the quadratic<br>
term embedded in it).  Try adding a third hidden neuron:</p>
<pre data-code-wrap="python"><code class="lang-python">		self.fc1 = nn.Linear (2, 3)
		self.fc2 = nn.Linear (3 ,1)
</code></pre>
<p>Please see this related post for some additional explanation:</p>
<aside class="quote quote-modified" data-post="1" data-topic="90913">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pemfir/48/43803_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/simple-pattern-is-not-detected-by-simple-linear-model/90913">Simple pattern is not detected by simple linear model</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    I have a very simple example of 6 training data with 6 labels. I am trying to fit a simple logistic regression to it, and hoping it should over-fit with 0 loss, but the model does not converge. 
Here is the rule, i am hoping the model to learn: 

When input is  [1., 0., 0., 1.] , the label is 1
When input is  [0., 1., 1., 0.] , the label is 1
When input is  [1., 0., 1., 0.] , the label is 0
When input is  [0., 1., 0., 1.] , the label is 0

is such pattern something a non-linear model cannot capt…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>