Neural Network : really high loss binary classification

username: vidivinc
cooked: <p>Hello everybody,<br>
I have an issues.<br>
I had a really unbalanced datased, I rebalanced it and after I applied a Neural Network.<br>
The problem is that it gives me high Loss and Low accuracy.<br>
I have already applied Kfold validation, standard scaler and also testsplit.<br>
My loss fuction is binary crossentropy because I have to do a binary classification.<br>
Can you help me?</p>

username: ptrblck
cooked: <p>Could you check, if your model is still overfitting, i.e. is it predicts just a single class?<br>
If so, could you post the model code as well as your training routine, so that we could check if for some errors?</p>

username: vidivinc
cooked: <p>At beginning problem was that class were umbalced and I balanced.<br>
Balancement works for f1 score in svc  , but none improvment from f1 score of neural netowrk, it also reduce the accuracy! why? . I also tried Kfold and crossfold validation, same result. Also tried to semplify level, but nothing.<br>
scaling:<br>
`</p>
<p>X_ = dataset_train_upsampled.drop([‘LABEL’], axis=1)<br>
y_ = dataset_train_upsampled[‘LABEL’]<br>
X_=np.array(X_)<br>
y_=np.array(y_)<br>
X_train_full_, X_test_, y_train_full_, y_test_ = train_test_split(X_, y_,test_size=0.2, random_state=42)<br>
X_train_, X_validation_, y_train_, y_validation_ = train_test_split(X_train_full_, y_train_full_, random_state=42)<br>
scaler = StandardScaler()</p>
<p>X_train_ = scaler.fit_transform(X_train_)</p>
<p>X_validation_ = scaler.transform(X_validation_)</p>
<p>X_test_ = scaler.transform(X_test_)<br>
`</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4bc25aa02eeb2ba4f9c30c1069ca07f2000a1eb2.png" data-download-href="https://discuss.pytorch.org/uploads/default/4bc25aa02eeb2ba4f9c30c1069ca07f2000a1eb2" title="reteneurale"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4bc25aa02eeb2ba4f9c30c1069ca07f2000a1eb2_2_690x309.png" alt="reteneurale" data-base62-sha1="aOcd8fg9vf9nax6ohgZHJLyqu0a" width="690" height="309" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4bc25aa02eeb2ba4f9c30c1069ca07f2000a1eb2_2_690x309.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4bc25aa02eeb2ba4f9c30c1069ca07f2000a1eb2_2_1035x463.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4bc25aa02eeb2ba4f9c30c1069ca07f2000a1eb2_2_1380x618.png 2x" data-dominant-color="353532"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">reteneurale</span><span class="informations">1479×663 47 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>The accuracy might be reduced for a balanced dataset due to the <a href="https://en.wikipedia.org/wiki/Accuracy_paradox">Accuracy Paradox</a>.<br>
If you are dealing with a highly imbalanced dataset, your model might just predict the majority class and thus get a high accuracy without “learning” anything. How does the F1 score look during training?</p>

username: vidivinc
cooked: <p>F1 score is 0 for the second class, and worse for the first-one. In generally it gets worse.<br>
I really can’t understand. I trained other models(SVC, Trees, Random forest), and f1-score improve significantly after upsampling<br>
I have problems only with neural network. Could be possible that neural network is a model too complicated to solve this problem?<br>
Dataset is umbalaced and training set is really poor.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/a/caf1b1c94ff29920dafa8aa54b6f115b9db9140a.png" alt="Cattura1" data-base62-sha1="sXkhb1yGYJe2dAPLrnEtoqIS4I2" width="593" height="180"> .</p>
<p>just to be clear, this is the result of neural network.</p>

username: vidivinc
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/0/80e2f11501540a394692bc0fede186ca7d3d9855.png" alt="Cattura1" data-base62-sha1="iobkRZ4RVcLt5S5G1JOp5EPdT3T" width="578" height="194"></p>
<p>this one of SVC.</p>
<p>both after upsamplig.</p>
<p>Really thanks</p>

username: ptrblck
cooked: <p>I’m not familiar with your Keras implementation (and you might get a better answer on StackOverflow <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> ).</p>
<p>That being said, I’m a bit confused by the output metric for your neural network.<br>
Your model seems to have one output neuron, which implies it should be able to predict two classes (positive and negative).<br>
How are you creating the result table and are you sure you are passing the expected format?</p>

username: vidivinc
cooked: <p>never mind ptrblck! really thank you for your time and your answer <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>