Huge False Negatives

hariram_manohar: <p>Hi,</p>
<p>Below are my test loss and validation loss</p>
<p>epoch: 300 loss: 0.0068271682<br>
Validation Loss: 0.00671717</p>
<p>But still i have false negatives completely, dont know where i am doing wrong.</p>
<p>confusion matrix:<br>
[[7450    0]<br>
[2318    0]]</p>
<pre><code>          precision    recall  f1-score   support

             0       0.76      1.00      0.87      7450
             1       0.00      0.00      0.00      2318

accuracy                                  0.76      9768
</code></pre>
<p>macro avg       0.38      0.50      0.43      9768<br>
weighted avg       0.58      0.76      0.66      9768</p>
<p>Accuracy:<br>
0.7626945126945127</p>
<p>Looks like other category in binary classification is completely treated as other(0/1), so thats why i got 2318 all are false negatives.<br>
i followed below blog for reference:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://stackabuse.com/introduction-to-pytorch-for-classification/">
  <header class="source">
      <img src="https://stackabuse.com/assets/images/favicon.ico" class="site-icon" width="" height="">

      <a href="https://stackabuse.com/introduction-to-pytorch-for-classification/" target="_blank" rel="noopener nofollow ugc" title="02:33PM - 21 October 2019">Stack Abuse – 21 Oct 19</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://stackabuse.com/introduction-to-pytorch-for-classification/" target="_blank" rel="noopener nofollow ugc">Introduction to PyTorch for Classification</a></h3>

  <p>PyTorch, alongside Tensorflow, is an extremely popular deep learning library for Python. In this article, we'll cover one of the basic tasks in machine learning - classification.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Sample dataset:</p>
<p>(48842, 15)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e657f4a0ea1fd870e6c3d5728066c89c9ae1119.png" data-download-href="https://discuss.pytorch.org/uploads/default/2e657f4a0ea1fd870e6c3d5728066c89c9ae1119" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e657f4a0ea1fd870e6c3d5728066c89c9ae1119.png" alt="image" data-base62-sha1="6Crq4ay77SlAoXPcEq9qlQtf1kB" width="690" height="53" data-dominant-color="060606"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1789×139 6.39 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>class is the predictable one.</p>

ptrblck: <p>Try to play around with some hyperparameters, e.g. the learning rate.<br>
Also, since your dataset is a bit imbalanced, you could try to add weights to your loss function.</p>

hariram_manohar: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, i will try to do that, but in the first place why would none of the other class not able to find a single prediction. how can i pinpoint exactly where its going wrong</p>

ptrblck: <p>Your model might be overfitting the majority class, which is common if you are dealing with an imbalanced dataset.<br>
In such a case the loss would be low, as e.g. 75% of your output is right by just predicting the majority class.<br>
I wouldn’t expect a constant prediction for a 25-75 imbalance, but it still seems to be a likely reason.</p>

hariram_manohar: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> i am trying to add weights for categorical target data, i have searched google but i didnt get anywhere exactly what i need. if you have any reference can you please mention to add weights to the categorical target along with embeddings.</p>
<p>Thanks.</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> accepts a <code>weight</code> argument as stated in the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss">docs</a>.<br>
If you are using <code>nn.BCEWithLogitsLoss</code>, you would use the <code>pos_weight</code> argument.</p>

hariram_manohar: <p>below is the code:</p>
<pre><code>weight = [3.17]
class_weight = torch.FloatTensor(weight)
loss_function = nn.BCEWithLogitsLoss(pos_weight=class_weight)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

y_pred = model(categorical_train_data, numerical_train_data)
print(y_pred.shape, train_outputs.shape)
single_loss = loss_function(y_pred, train_outputs)
#single_loss = loss_function(y_pred, torch.max(train_outputs,1)[1])
</code></pre>
<p>output:</p>
<pre><code>torch.Size([39074, 2]) torch.Size([39074, 1])
  File "/apps/miniconda3/envs/torch/lib/python3.8/site-packages/torch/nn/functional.py", line 2124, in binary_cross_entropy_with_logits
raise ValueError("Target size ({}) must be the same as input size ({})".format(target.size(), input.size()))
 ValueError: Target size (torch.Size([39074, 1])) must be the same as input size (torch.Size([39074, 2]))
</code></pre>
<p>Just checking what to do</p>

hariram_manohar: <p>what should i do? should i reduce my input y_pred to 1 dimension ?</p>

ptrblck: <p>Based on the target shape it seems you are dealing with a binary classification.<br>
If you are using <code>nn.BCEWithLogitsLoss</code>, then your model would have to output a single logit, while it seems to be returning currently 2 outputs.</p>

hariram_manohar: <p>okay… is it the same scenario in multi-classification too ? if you have 0-9 then output will be single logit only.</p>

hariram_manohar: <p>this blog tutorial also showed binary classification, but they have updated output_size as 2, does they mentioned wrong ?</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://stackabuse.com/favicon.png" class="site-icon" width="16" height="16">
      <a href="https://stackabuse.com/introduction-to-pytorch-for-classification/" target="_blank" title="02:33PM - 21 October 2019" rel="nofollow noopener">Stack Abuse – 21 Oct 19</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="16" height="16">

<h3><a href="https://stackabuse.com/introduction-to-pytorch-for-classification/" target="_blank" rel="nofollow noopener">Introduction to PyTorch for Classification</a></h3>

<p>PyTorch [https://pytorch.org/]  and TensorFlow [https://www.tensorflow.org/] 
libraries are two of the most commonly used Python libraries for deep learning.
PyTorch is developed by Facebook, while TensorFlow is a Google project. In this
article, you...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


hariram_manohar: <p>if the things mentioned in blog is true, should i go for opt the solution 2 mentioned in below site (reshape the input tensor from n.m to n.l)</p>
<p><a href="http://www.programmersought.com/article/1886850916/" class="onebox" target="_blank" rel="nofollow noopener">http://www.programmersought.com/article/1886850916/</a></p>

ptrblck: <p>For a binary classification you could use:</p>
<ul>
<li>one output value and use <code>nn.BCEWithLogitsLoss</code>
</li>
<li>or two outputs and use <code>nn.CrossentropyLoss</code>
</li>
</ul>
<p>Both approaches would work, but are a bit different in their implementation.<br>
While the former is used for a classical binary classification as well as a multi-label classification (each sample might belong to zero, one, or multiple classes), the latter is used for a multi-class classification (each sample belongs to one class only).</p>

hariram_manohar: <p>Okay, Thanks i will try to BCEWithLogitsLoss as mine in binary classification, will try to update output_size as 1 and will try to change obtained tensor from model to int from float as target is type of int.</p>

hariram_manohar: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>tried all the possibilities, even followed the suggestion given by <a class="mention" href="/u/done1892">@done1892</a> from <a href="https://discuss.pytorch.org/t/what-is-the-weight-values-mean-in-torch-nn-crossentropyloss/11455/10" class="inline-onebox">What is the weight values mean in torch.nn.CrossEntropyLoss? - #10 by done1892</a> for BCEwithlogitsloss.</p>
<p>This is my classification ratio in training set.</p>
<blockquote>
<p>&lt;=50K    29705<br>
(greater than)50K      9369<br>
Name: class, dtype: int64</p>
</blockquote>
<p>so i have tried with pos_ratio of 3.17 and 1/3.17 too… but still none of the confusion_matrix or accuracy didnt changed a bit.</p>
<p>Here is my code:</p>
<pre><code>weights = [[2.5],[2.55],[2.6],[2.65],[2.7],[2.75],[2.8],[2.85],[2.9],[2.95],[3],[3.05],[3.1],[3.15],[3.2],[3.25],[3.3],[3.35],[3.4],[3.45],[3.5],[3.55],[3.6],[3.65],[3.7],[3.75],[3.8],[3.85],[3.9],[3.94999999999999],[3.99999999999999],[4.04999999999999],[4.09999999999999],[4.14999999999999],[4.19999999999999],[4.24999999999999],[4.29999999999999],[4.34999999999999],[4.39999999999999],[4.44999999999999],[4.49999999999999],[4.54999999999999],[4.59999999999999],[4.64999999999999],[4.69999999999999],[4.74999999999999],[4.79999999999999],[4.84999999999999],[4.89999999999999],[4.94999999999999],[4.99999999999999],[5.04999999999999],[5.09999999999999],[5.14999999999999],[5.19999999999999],[5.24999999999999],[5.29999999999999],[5.34999999999999],[5.39999999999999],[5.44999999999999],[5.49999999999999],[5.54999999999999],[5.59999999999999],[5.64999999999999],[5.69999999999999],[5.74999999999999],[5.79999999999999],[5.84999999999999],[5.89999999999999],[5.94999999999999],[5.99999999999999],[6.04999999999999],[6.09999999999999],[6.14999999999999],[6.19999999999999],[6.24999999999999],[6.29999999999999],[6.34999999999999],[6.39999999999999],[6.44999999999999],[6.49999999999999],[6.54999999999999],[6.59999999999999],[6.64999999999999],[6.69999999999999],[6.74999999999998],[6.79999999999998],[6.84999999999998],[6.89999999999998],[6.94999999999998],[6.99999999999998],[7.04999999999998],[7.09999999999998],[7.14999999999998],[7.19999999999998],[7.24999999999998],[7.29999999999998],[7.34999999999998],[7.39999999999998],[7.44999999999998],[7.49999999999998],[7.54999999999998],[7.59999999999998],[7.64999999999998],[7.69999999999998],[7.74999999999998],[7.79999999999998],[7.84999999999998],[7.89999999999998],[7.94999999999998],[7.99999999999998],[8.04999999999998],[8.09999999999998],[8.14999999999998],[8.19999999999998],[8.24999999999998],[8.29999999999998],[8.34999999999998],[8.39999999999998],[8.44999999999998],[8.49999999999998],[8.54999999999998],[8.59999999999998],[8.64999999999998],[8.69999999999998],[8.74999999999998],[8.79999999999998],[8.84999999999998],[8.89999999999998],[8.94999999999998],[8.99999999999998],[9.04999999999998],[9.09999999999998],[9.14999999999998],[9.19999999999998],[9.24999999999998],[9.29999999999998],[9.34999999999998],[9.39999999999998],[9.44999999999998],[9.49999999999998],[9.54999999999997],[9.59999999999997],[9.64999999999997],[9.69999999999997],[9.74999999999997],[9.79999999999997],[9.84999999999997],[9.89999999999997],[9.94999999999997],[9.99999999999997],[10.05],[10.1],[10.15],[10.2],[10.25],[10.3],[10.35],[10.4],[10.45],[10.5],[10.55],[10.6],[10.65],[10.7],[10.75],[10.8],[10.85],[10.9],[10.95],[11],[11.05],[11.1],[11.15],[11.2],[11.25],[11.3],[11.35],[11.4],[11.45],[11.5],[11.55],[11.6],[11.65],[11.7],[11.75],[11.8],[11.85],[11.9],[11.95],[12],[12.05],[12.1],[12.15]
]

rate = [0.001,0.005,0.009,0.013,0.017,0.021,0.025,0.029,0.033,0.037,0.041,0.045,0.049,0.053,0.057,0.061,0.065,0.069,0.073,0.077]

for lr in rate:
	for wg in weights:
		#Create Model object
		model = Model(categorical_embedding_sizes, numerical_data.shape[1], 1, [512,128,16], p=0.4)
		
		#Create loss and optimizer function

		#Add Weights
		#weights = [3.17,1.0]
		#class_weights = torch.FloatTensor(weights)

		#loss_function = nn.CrossEntropyLoss(weight=class_weights)
		class_weight = torch.FloatTensor(wg)
		loss_function = nn.BCEWithLogitsLoss(pos_weight=class_weight)
		optimizer = torch.optim.Adam(model.parameters(), lr=0.001)



		#Train the model

		epochs = ep
		aggregated_losses = []

		for i in range(epochs):
			i += 1
			y_pred = model(categorical_train_data, numerical_train_data)
			#print(y_pred[:5])
			#print(y_pred.shape, train_outputs.shape)
			#out, inds = torch.max(y_pred,dim=1)
			#out = out.unsqueeze(1)
			train_outputs = train_outputs.float()
			single_loss = loss_function(y_pred, train_outputs)
			#single_loss = loss_function(y_pred, torch.max(train_outputs,1)[1])
			aggregated_losses.append(single_loss)

			#if i%25 == 1:
			#	print(f'epoch: {i:3} loss: {single_loss.item():10.8f}')

			optimizer.zero_grad()
			single_loss.backward()
			optimizer.step()

		print(f'epoch: {i:3} loss: {single_loss.item():10.10f}')


		#plt.plot(range(epochs), aggregated_losses)
		#plt.ylabel('Loss')
		#plt.xlabel('epoch');




		##Make Predictions
		with torch.no_grad():
			y_val = model(categorical_test_data, numerical_test_data)
			#out, inds = torch.max(y_val,dim=1)
			#out = out.unsqueeze(1)
			test_outputs = test_outputs.float()
			loss = loss_function(y_val, test_outputs)
			#loss = loss_function(y_val, torch.max(test_outputs,1)[1])
		print(f'Validation Loss: {loss:.8f}')


		#Convert output values to either 0 or 1
		y_val = np.argmax(y_val, axis=1)
		#print(y_val[:5])


		print(confusion_matrix(test_outputs,y_val))
		#print(classification_report(test_outputs,y_val))
		print(accuracy_score(test_outputs, y_val))
</code></pre>
<p>please take a look and advice.</p>

hariram_manohar: <p>FYI<br>
i tried with catboost and with simple tuning i am able to achieve good results…i dont understand why this much difficult with DNN using pytorch.</p>
<pre><code>    Test data:
 &lt;=50K    7451
 &gt;50K     2318
Name: class, dtype: int64
 Confusion matrix:
[[7037  799]
 [ 414 1519]]
</code></pre>
<p>Here we got 799 FN’s and 414 FP’s</p>

hariram_manohar: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> or any one suggest please</p>

hariram_manohar: <p>Hi,</p>
<p>even weights are not helping out for imbalanced data… please let me know how to do it.</p>

ptrblck: <p>Using a weighted loss usually trades the metrics, e.g. FP for FN.<br>
Are you not seeing any change in the confusion matrix using a weighted loss?</p>

hariram_manohar: <p>no, not at all. wondered to see no changes… as per thumb rule i have assigned weight (lowest/highest)</p>
<pre><code>weight = [0.317]
class_weight = torch.FloatTensor(weight)
loss_function = nn.BCEWithLogitsLoss(pos_weight=class_weight)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
</code></pre>
<p>Here is the classification groupings:</p>
<p>traindata:</p>
<pre><code>          &lt;=50K    29705
          &gt;50K      9369
</code></pre>
<p>testdata:</p>
<pre><code> &lt;=50K    7450
 &gt;50K     2318
</code></pre>
<p>assigned weight as (9369/29705) and got CM as it is as if no weights are assigned:</p>
<pre><code>epoch: 300 loss: 0.3204252720
Validation Loss: 0.31998613
[[7450    0]
 [2318    0]]

			  precision    recall  f1-score   support

		 0.0       0.76      1.00      0.87      7450
		 1.0       0.00      0.00      0.00      2318

	accuracy                           0.76      9768
   macro avg       0.38      0.50      0.43      9768
weighted avg       0.58      0.76      0.66      9768

0.7626945126945127</code></pre>