ROC-AUC is high but PR-AUC value is very low

ina: <p>Hello,</p>
<p>I am working on DNA sequences data and using CNN. My dataset is hugely imbalanced.<br>
positive class samples (~500)<br>
negative class samples (~150,000)</p>
<p>So I am using <code>WeightedRandomSampler</code> to oversample and balance classes before feeding to data loader.</p>
<p>I use a 5-fold cross-validation. When I did few test runs, I could get a decent ROC value but the PR-AUC value seems to be really low.</p>
<p>For fold 1:<br>
roc auc 0.9667848699763594<br>
precision auc 0.055329116326074484</p>
<p>For fold 2:<br>
roc auc 0.8476321207961566<br>
precision auc 0.03307627288669479</p>
<p>For fold 3:<br>
roc auc 0.9528898540612085<br>
precision auc 0.05020178518546394</p>
<p>I suspect that there are lot of false negatives. Since the positive class samples (~500) is very low compared to negative class samples (~150,000) the model learns the negative class better and predicts most of the test samples as negative.</p>
<p>I tried weighing the positive class using<br>
<code>weight = [50.0]</code><br>
<code>class_weight = torch.FloatTensor(weight).to(device)</code><br>
<code>criterion = nn.BCEWithLogitsLoss(pos_weight=class_weight)</code><br>
By doing this, almost all samples are predicted as positive.</p>
<p>I tried Adaptive learning rates as well but the precision-recall values do not seem to improve.<br>
<strong>Can someone guide me and let me know the ideas to improve Precision and Recall values?</strong></p>
<p>Thanks!</p>

ptrblck: <p>That’s generally a tough problem. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
Could you post the confusion matrix, so that we get a feeling about the predictions?</p>

ina: <p>I understand <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_frowning_face.png?v=9" title=":slightly_frowning_face:" class="emoji" alt=":slightly_frowning_face:"> :<br>
Since the imbalance is too high, the model predicts most of the samples as negatives (class 0)<br>
A sample of my confusion matrix:<br>
[[1023    0]<br>
[   1    0]]</p>
<p>[[1022    0]<br>
[   2    0]]</p>
<p>[[1018    0]<br>
[   6    0]]</p>

ptrblck: <p>I would try to lower the impact of the imbalance a bit and subsample the negative samples before training to e.g. 15000 samples (or even lower), and then retry the <code>WeightedRandomSampler</code>.</p>
<p>Btw. could you post the code snippet you are using to create the class counts, weights, and <code>WeightedRandomSampler</code>?<br>
I would like to make sure nothing goes wrong there, as your model still overfits badly on the negative class.</p>

ina: <p>Yeah sure.</p>
<pre><code class="lang-auto">class_sample_count = np.array([len(np.where(Ytr == t)[0]) for t in np.unique(Ytr)])
weight = 1. / class_sample_count
samples_weight = np.array([weight[t] for t in Ytr])
samples_weight = torch.from_numpy(samples_weight)
sampler = torch.utils.data.sampler.WeightedRandomSampler(samples_weight.type('torch.DoubleTensor'), len(samples_weight))
train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=batch_size, num_workers=1, sampler=sampler)
</code></pre>
<p>Maybe I will try to under sample negative sets and try to see how it behaves.</p>

ptrblck: <p>The code looks alright. Let me know, how the experiments worked out.</p>

ina: <p>Sure <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Thank you for your response</p>

ina: <p>Hello!</p>
<p>I did the downsampling way too. The results are still the same <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

ptrblck: <p>Ok, thanks for the information.<br>
Let’s maybe try to scale down the problem and have a minimal working version.<br>
Could you post the model definition, so that we could use it as a starter?<br>
If that’s not possible due to licensing etc., feel free to create a “similar” dummy model.</p>

ina: <p>Yeah sure. Thanks for your help.<br>
Here it is</p>
<pre><code class="lang-auto">class ClDataset(Dataset):
  def __init__(self, X, Y):
        self.len = len(X)
        temp = np.asarray(X, np.float64)
        self.x = torch.from_numpy(temp)
        self.y = torch.from_numpy(np.asarray(Y))

  def __getitem__(self, index):
        return self.x[index], self.y[index]

  def __len__(self):
        return len(self.y)


class ClCNN(nn.Module):

    def __init__(self):
        super(ClCNN, self).__init__()
        # convolutional layer
        self.layer11 = nn.Sequential(
            nn.Conv2d(1, 8, kernel_size=(4, 3), stride=1, padding=(2, 2)),
            nn.BatchNorm2d(8),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))

        self.layer12 = nn.Sequential(
            nn.Conv2d(1, 8, kernel_size=(4, 5), stride=1, padding=(2, 2)),
            nn.BatchNorm2d(8),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))

        self.layer21 = nn.Sequential(
            nn.Conv2d(8, 16, kernel_size=(3, 2), stride=1, padding=(2, 2)),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=1, stride=1))

        self.layer22 = nn.Sequential(
            nn.Conv2d(8, 16, kernel_size=(3, 3), stride=1, padding=(2, 2)),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=1, stride=1))

        self.fc1 = nn.Linear(6944, 512)
        self.fc2 = nn.Linear(512, 2)

    def forward(self, x):
        x = x.float()
        out11 = self.layer11(x)
        out12 = self.layer12(x)

        out21 = self.layer21(out11)
        out21 = out21.reshape(out21.size(0), -1)
        out22 = self.layer22(out12)
        out22 = out22.reshape(out22.size(0), -1)
        
        out = torch.cat((out21, out22), 1)
        out = self.fc1(out)
        out = F.dropout(out, p=0.5, training=self.training)
        out = F.log_softmax(self.fc2(out), dim=-1)

        return out
</code></pre>

ina: <p>I am using SGD optimizer and CrossEntropyLoss with lr=0.001.</p>
<p>Maybe I should try to tune my hyperparameters. Is it possible for you to share some example of how to use Hyperopt or Hypersearch in Pytorch CNN??</p>

ptrblck: <p>Thanks for the code!<br>
<code>nn.CrossEntropyLoss</code> applies <code>F.log_softmax</code> and <code>nn.NLLLoss</code> internally, so could you please remove the <code>F.log_softmax</code> from your model or use <code>nn.NLLLoss</code> as the criterion, and rerun the experiment?</p>

ina: <p>Aah ok. I am just running it. Will update you in some time. Thank you for your suggestions so far <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ina: <p>Did 2 runs so far and the results are definitely better <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>roc auc 0.8597332329593886<br>
precision auc 0.1152646061940882</p>
<p>roc auc 0.8950204024201491<br>
precision auc 0.29831894063835274</p>
<p>Thank you so much!!</p>

ptrblck: <p>Puh, I was running out of ideas and couldn’t believe that the model is overfitting that much even with weighted sampling. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
Feel free to post updates on your experiments, as I’m always interested in these imbalanced cases. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ina: <p>Sure <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I got this using under sampling. I will also try using WeightedRandomSampler and let you know <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Thank you once again !!</p>

ina: <p>Do you think using a hyperparameter optimization technique can improve the results? I am reading about different techniques available like hypersearch and hyperopt but since I am new to DL I am not able to get a grip of it. Is it posible for you to post any sample implementation of hypersearch or hyperopt for pytorch CNN?</p>

ptrblck: <p>These techniques might help, but I’m unfortunately inexperienced in this topic and just played around with some architecture search methods. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"> So let’s better wait for some experts and their opinion. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ina: <p>Oh Okay sure <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ina: <p>Hey!! Happy Christmas <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Hope your Christmas is going good!!</p>
<p>Incase you are working this week, I would just like to post a question. I managed to get the PR-AUC to say like around 23%. Do you have any ideas to increase it more?? Something like for example</p>
<ul>
<li>
<p>making the network more complex or simple</p>
</li>
<li>
<p>or increasing the batch size etc.</p>
</li>
</ul>