How can I get better performance from a multi-label image classifier?

JoshP: <p>Hi, I am trying to create a CNN to classify images with multiple labels.</p>
<p>Labels are of the form:</p>
<p>[0., 0., 0., 0., 0., 0., 0., 0., 0.5000, 0., 0., 0., 0., 0., 1., 0., 0., 0.,1., 0.]</p>
<p>where 0 represents no appearance of the category, 0.5 represents an appearance that is hard to identify and 1 represents a clear appearance of the category in the image.</p>
<p>The data is imbalanced:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/b/cbabf1af5cf78cda8c96b3527e80b5fa3d920c9d.png" data-download-href="https://discuss.pytorch.org/uploads/default/cbabf1af5cf78cda8c96b3527e80b5fa3d920c9d" title="image_2020-11-04_110316"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/b/cbabf1af5cf78cda8c96b3527e80b5fa3d920c9d.png" alt="image_2020-11-04_110316" data-base62-sha1="t3LjwQQD5ESGGWhK9emQizVbLY1" width="690" height="327" data-dominant-color="ECE4EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image_2020-11-04_110316</span><span class="informations">957×454 15.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have tried training with all categories and also with the ‘person’ category removed but in both examples the model performs fairly poorly either by nearly always placing the highest probability on ‘person’ or if person is removed it tends to place the highest probability on the same few categories regardless of image:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/4/b4eadd1a8808d8b52d40f8cb62e18b889866ed37.png" data-download-href="https://discuss.pytorch.org/uploads/default/b4eadd1a8808d8b52d40f8cb62e18b889866ed37" title="image_2020-11-04_111831"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b4eadd1a8808d8b52d40f8cb62e18b889866ed37_2_415x500.png" alt="image_2020-11-04_111831" data-base62-sha1="pOt8xJoWbCNEYtAMYCR1DMcFvkH" width="415" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b4eadd1a8808d8b52d40f8cb62e18b889866ed37_2_415x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b4eadd1a8808d8b52d40f8cb62e18b889866ed37_2_622x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/4/b4eadd1a8808d8b52d40f8cb62e18b889866ed37.png 2x" data-dominant-color="EDEDEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image_2020-11-04_111831</span><span class="informations">647×778 118 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I semi adapted the network layers from a tutorial for binary classification but I think the design is not optimal, I am hoping someone can give me some advice on a better network design. This is what I am currently using:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/b/5bd4a8daae3ce79690b8aac33e842745b6bbe7a2.png" data-download-href="https://discuss.pytorch.org/uploads/default/5bd4a8daae3ce79690b8aac33e842745b6bbe7a2" title="image_2020-11-04_111139"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/b/5bd4a8daae3ce79690b8aac33e842745b6bbe7a2.png" alt="image_2020-11-04_111139" data-base62-sha1="d6n4orkextufgdEkFQqOkewrGr8" width="690" height="491" data-dominant-color="F0F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image_2020-11-04_111139</span><span class="informations">847×603 27.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>and this is my training loop:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/0/20c380c703162077e897a3d9c735fe46af1520ce.png" data-download-href="https://discuss.pytorch.org/uploads/default/20c380c703162077e897a3d9c735fe46af1520ce" title="image_2020-11-04_111254"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/0/20c380c703162077e897a3d9c735fe46af1520ce.png" alt="image_2020-11-04_111254" data-base62-sha1="4FQ8PiPiTe6SO6gZLYFtzNdi6KO" width="373" height="500" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image_2020-11-04_111254</span><span class="informations">664×890 28 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am also unsure of the best way to evaluate the model on the validation set. Currently I am calculating the euclidean distance between the output and label vector and calculating the total mean distance but I feel like there should be a better way for multi-label data.</p>

anujd9: <p>Hi. There are a couple of things you can use here:</p>
<ol>
<li>
<p>If you want to use this model, you can try out. using data augmentation using transforms for training you model on more images and its variations. Additionally, you should increase the number of filters for your Conv2d layers that should help the model to learn more types of underlying features and train better. For example, increasing the out_channels in the first Conv2d layer from 12 to something like 64 and similarly for the rest of the layers. Usually the best practice is to make sure that these are powers of 2 i.e. 16, 32, 64, 128 and so on.</p>
</li>
<li>
<p>Instead of using this architecture, you can make use of pre-trained networks and use the technique of Transfer Learning. That should give you a pretty good accuracy.</p>
</li>
<li>
<p>Additionally, play around with different optimizers like Adam, Adadelta, RMSProp etc.</p>
</li>
</ol>
<p>I hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

JoshP: <p>Hi, thanks for the advice. I increased the cov2d layer sizes and set the first layers kernel size to 3 and set the optimiser to Adam which seems to have increased performance.</p>
<p>I agree that for this task transfer learning would yield better results but I wanted to try and implement and train my own network.</p>

anujd9: <p>I am glad that this helped. If you are all good, can you please mark the above comment as the solution so that others with a same issue can also get some help from that ? Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>