Lung nodule classification - High imbalance

rphellan: <p>Hello, I am working on a lung nodule classification problem, as either benign or malignant. The inputs are CT volumes in 3D. We have segmentations for each nodule, so all other tissues around the nodule are removed.</p>
<p>The dataset is highly imbalanced, with thousands of benign nodules samples, and only a few hundred malignant samples. I am using the WeightedRandomSampler, as it is suggested in other posts. I also added weights to the cross-entropy loss function (3.0 for the positive class: malignant). The function used to calculate the validation accuracy is the F beta score, with weight 0.5, to give more weight to the recall. The model used is DenseNet 121.</p>
<p>In general, I see the loss function going down, but the recall of the model in the test set goes down too, as the epochs progress. I was hoping for the opposite, to have a higher test recall. Do you have any suggestions? I can give more details if needed.</p>

KFrank: <p>Hi Renzo!</p>
<aside class="quote no-group" data-username="rphellan" data-post="1" data-topic="189182" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/0ea827/48.png" class="avatar"> rphellan:</div>
<blockquote>
<p>only a few hundred malignant samples.</p>
</blockquote>
</aside>
<p>Depending on the details of your use case and the complexity of your model,<br>
a few hundred positive samples might not be enough to train effectively.  (You<br>
should be able to overfit and get “good” results on your training set, without<br>
necessarily getting good results on your validation set.)</p>
<p>If this is the case (and you can’t get more data), you should consider trying<br>
<em>data augmentation,</em> where you make additional semi-synthetic input data by<br>
doing “irrelevant” things like stretching or cropping or rotating your real images.<br>
This can permit you to train longer on “more” data, while reducing the risk of<br>
overfitting.</p>
<aside class="quote no-group">
<blockquote>
<p>I am using the WeightedRandomSampler,</p>
</blockquote>
</aside>
<p>This would be my preferred approach to an unbalanced dataset, assuming<br>
that you have enough positive samples (relative to your batch size) that it is<br>
unlikely for a given batch to contain duplicate positive samples.</p>
<aside class="quote no-group">
<blockquote>
<p>I also added weights to the cross-entropy loss function</p>
</blockquote>
</aside>
<p>This is also a reasonable approach (although I would lean against using<br>
both <code>WeightedRandomSampler</code> and a loss-function weight, if only to make<br>
the training process easier to reason about).</p>
<aside class="quote no-group">
<blockquote>
<p>(3.0 for the positive class: malignant).</p>
</blockquote>
</aside>
<p>I’m curious how you came up with 3.0 for your positive-class weight.</p>
<aside class="quote no-group">
<blockquote>
<p>In general, I see the loss function going down, but the recall of the model in the test set goes down too, as the epochs progress.</p>
</blockquote>
</aside>
<p>Greater context would be helpful.  I would suggest that you look at graphs, as<br>
a function of training epoch, of not only the recall for your training set, but the<br>
loss, recall, and accuracy (and any other performance metrics you especially<br>
care about, such as your F-beta score), for both your training and validation<br>
datasets.  The idea would be to see how all of these evolve as you train, look<br>
for any evidence of overfitting, and so on.</p>
<p>In particular, if your validation-set loss function is going down nicely, but your<br>
validation-set performance metrics are also degrading, you would want to<br>
understand why you have a disconnect between your loss function and your<br>
performance metrics.</p>
<p>Best.</p>
<p>K. Frank</p>

rphellan: <p>Thanks, KFrank. That is very detailed advise.</p>
<p>As a summary, the WeightedRandomSampler gave me good results. I tried some data augmentation but it didn’t really help to increase the performance. Also, I removed the loss-function weights, as I did not see any major change. The 3.0 weight was just a guess.</p>
<p>An interesting fact when I plotted the loss-function vs validation is that it took very few epochs, around 3 to 6, to reach the highest values. Then, early-stopping was very helpful.</p>
<p>My guess is that because I use already segmented lung nodules (around 10-20% of the image is the nodule and 80-90% is just uniform background), there is not much data to process, and it doesn’t take too many epochs to fit the model.</p>
<p>I could get around 84 - 86 % accuracy, sensitivity, and specificity, which is state-of-the-art for the DenseNet. I plan to try other architectures, like Vision Transformers, which seem to generate more accurate results.</p>

KFrank: <p>Hi Renzo!</p>
<aside class="quote no-group" data-username="rphellan" data-post="3" data-topic="189182" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/0ea827/48.png" class="avatar"> rphellan:</div>
<blockquote>
<p>I tried some data augmentation but it didn’t really help to increase the performance.</p>
</blockquote>
</aside>
<p>My view of data augmentation is not that – all else being equal – it will lead<br>
to better performance.  Rather, using data augmentation will hopefully delay<br>
the onset of overfitting so that you can achieve better performance by training<br>
longer (without overfitting).</p>
<p>If your goal is to achieve the best performance metrics you can (and you have<br>
sufficient computing resources) I would suggest using data augmentation and<br>
training until overfitting clearly sets in.</p>
<aside class="quote no-group">
<blockquote>
<p>it took very few epochs, around 3 to 6, to reach the highest values.</p>
</blockquote>
</aside>
<p>Six epochs seems very small to me, especially given that an epoch only contains<br>
a few hundred positive samples.  (Six epochs seems small even for fine tuning a<br>
pre-trained model.)</p>
<p>I would suggest that you try training significantly longer so that you can get a feel<br>
for how your loss and performance metrics bounce around from epoch to epoch.<br>
You don’t want to mistake epoch-to-epoch “noise” for your training starting to level<br>
off.  If your loss curve does start to level off (taking into consideration observed<br>
“noise”), but you don’t yet see clear signs of overfitting, keep training.  Sometimes<br>
you can get “stuck” on a plateau, but with significant additional training you can<br>
get unstuck and your model can continue to improve.  (And try data augmentation<br>
to delay overfitting as much as possible.)</p>
<p>Best.</p>
<p>K. Frank</p>

rphellan: <p>Thank you, <a class="mention" href="/u/kfrank">@KFrank</a>.</p>
<p>I decided to use weights and biases to plot the progress of the training. As it can be seen below, the loss goes down together with the recall, while the specificity and accuracy go up.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f91c21a44323963fadfff399d3834e927bd466b3.png" data-download-href="https://discuss.pytorch.org/uploads/default/f91c21a44323963fadfff399d3834e927bd466b3" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f91c21a44323963fadfff399d3834e927bd466b3_2_690x281.png" alt="image" data-base62-sha1="zxJa0tQgU8sa6ULyUom1ap0tlPd" width="690" height="281" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f91c21a44323963fadfff399d3834e927bd466b3_2_690x281.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f91c21a44323963fadfff399d3834e927bd466b3_2_1035x421.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f91c21a44323963fadfff399d3834e927bd466b3_2_1380x562.png 2x" data-dominant-color="FCFDFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1533×626 54.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am interested in optimising the average of recall and specificity, but it reaches a peak around epoch 17. Has this happened to you before? Any strategy you could suggest for this?</p>
<p>I plan to try dropout to delay overfitting, as you suggested, but I am also wondering if my loss function (torch.nn.BCEWithLogitsLoss) is not the right one.</p>

KFrank: <p>Hi Renzo!</p>
<aside class="quote no-group" data-username="rphellan" data-post="5" data-topic="189182" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/0ea827/48.png" class="avatar"> rphellan:</div>
<blockquote>
<p>I decided to use weights and biases to plot the progress of the training.</p>
</blockquote>
</aside>
<p>I don’t know what you mean by this.</p>
<aside class="quote no-group">
<blockquote>
<p>As it can be seen below, the loss goes down together with the recall, while the specificity and accuracy go up.</p>
</blockquote>
</aside>
<p>With the exception of your (training) loss curve, these all seem sufficiently<br>
noisy so as to prevent any conclusions from being drawn.</p>
<p>Your graph titled “val loss, loss” (the upper-left graph in the set of six) looks<br>
quite odd.  What appears to be your training loss is nice and smooth, while<br>
your “val loss” is very noisy.  Is your validation dataset much smaller than<br>
your training dataset, or otherwise significantly different in character?</p>
<aside class="quote no-group">
<blockquote>
<p>it reaches a peak around epoch 17.</p>
</blockquote>
</aside>
<p>There is so much epoch-to-epoch noise in your “average rec spec” graph<br>
that you can’t really conclude that such a peak is meaningful.</p>
<aside class="quote no-group">
<blockquote>
<p>I plan to try dropout to delay overfitting,</p>
</blockquote>
</aside>
<p>Assuming you manage to get your noise issue under control, you might want<br>
to reconsider data augmentation as another tool to delay overfitting.</p>
<aside class="quote no-group">
<blockquote>
<p>I am also wondering if my loss function (torch.nn.BCEWithLogitsLoss) is not the right one.</p>
</blockquote>
</aside>
<p>My understanding is that you do have a binary-classification problem: either<br>
benign or malignant.  Assuming that your prediction is a <em>logit,</em> typically the<br>
output of a final <code>Linear</code> layer with <code>out_features = 1</code> and no subsequent<br>
“activations” such as <code>sigmoid()</code>, then yes, <code>BCEWithLogitsLoss</code> will be an<br>
appropriate loss criterion.</p>
<p>Best.</p>
<p>K. Frank</p>