Image rotation detection

grobelny: <p>I wanted to create a model that would detect whether image is rotated 0, 90, 180 or 270 degrees (classify into one of 4 classes).</p>
<p>I took the following approach based on <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a> :</p>
<ol>
<li>Took about 400 self made photos, rescaled to minimum dimension of 256 pixels, took random crop of 224x224, random rotation (affects label), added flop (does not affect label). Then the dimensions are swapped, the pixel values are divided by 256 (to be in 0-1 range) and normalized (to be centered around 0).</li>
<li>Used resnet18 as base model and added fc layer with 4 outputs.</li>
<li>I am trying to teach my network something for 25 epochs, which on a CPU is quite time consuming <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">
</li>
</ol>
<p>Unfortunately while the training accuracy goes slightly above 90% the validation accuracy does not show the model is useful at all. I tried to check how much the model deviates from expected output “by hand” and it seems that the model does not depend much on the input - when looking at values assigned to a class on output they are very similar across validation images (eg. -0.2, 0.6, -0.4, 0.1).</p>
<p>Therefore I have the following questions:</p>
<ol>
<li>Do you think the approach makes sense for this scenario?</li>
<li>Do you think I have enough training data?</li>
<li>How can I debug my training/validation process? What can go wrong?</li>
</ol>

KFrank: <p>Hi Tomasz!</p>
<aside class="quote no-group quote-modified" data-username="grobelny" data-post="1" data-topic="56249" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/f07891/48.png" class="avatar"> grobelny:</div>
<blockquote>
<p>I wanted to create a model that would detect whether image is rotated 0, 90, 180 or 270 degrees (classify into one of 4 classes).</p>
<p>…</p>
<ol>
<li>Took about 400 self made photos, rescaled to minimum dimension of 256 pixels, took random crop of 224x224, random rotation (affects label), added flop (does not affect label). Then the dimensions are swapped, the pixel values are divided by 256 (to be in 0-1 range) and normalized (to be centered around 0).</li>
</ol>
</blockquote>
</aside>
<p>From your original 400 photos, how many derived photos<br>
did you make?  How large is your total dataset, that is, how<br>
many total images do you have, original plus augmented?</p>
<p>How do you split your dataset into training and validation?</p>
<p>Do you purposely include all four rotations of an image in<br>
your dataset (or all eight rotations plus reflection (flop))?<br>
(I would.)</p>
<p>Do you make a point of not putting augmented images derived<br>
from the same original image in both the training and validation<br>
datasets?  (Note, depending on the details of which augmented<br>
images went where this could arguably make your validation<br>
accuracy come out either better or worse.)</p>
<aside class="quote no-group">
<blockquote>
<ol start="2">
<li>Used resnet18 as base model and added fc layer with 4 outputs.</li>
</ol>
</blockquote>
</aside>
<p>I don’t know anything about resnet, but on the surface, this<br>
seems reasonable.</p>
<aside class="quote no-group">
<blockquote>
<ol start="3">
<li>I am trying to teach my network something for 25 epochs, which on a CPU is quite time consuming <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></li>
</ol>
<p>Unfortunately while the training accuracy goes slightly above 90% the validation accuracy does not show the model is useful at all. I tried to check how much the model deviates from expected output “by hand” and it seems that the model does not depend much on the input - when looking at values assigned to a class on output they are very similar across validation images (eg. -0.2, 0.6, -0.4, 0.1).</p>
</blockquote>
</aside>
<p>To me such numbers look like a reasonably strong prediction that<br>
your sample is in the second of your four classes (that is, the<br>
class with the predicted logit of 0.6).</p>
<p>What do such numbers look like for typical training images?</p>
<aside class="quote no-group">
<blockquote>
<p>Therefore I have the following questions:</p>
<ol>
<li>Do you think the approach makes sense for this scenario?</li>
</ol>
</blockquote>
</aside>
<p>This seems reasonable.</p>
<aside class="quote no-group">
<blockquote>
<ol start="2">
<li>Do you think I have enough training data?</li>
</ol>
</blockquote>
</aside>
<p>Quite possible not.  How many parameters are in your whole<br>
network?  How many are in your added fully-connected layer?</p>
<p>I’m speculating that resnet18 has millions of parameters.<br>
Although you augment your data, you start with 400 original<br>
samples (so maybe 200 original samples in your training set).<br>
I don’t know for certain, but it feels like a lot of parameters<br>
for about 200 samples.</p>
<p>Do you further train the whole network, or just the added layer?<br>
If you are further training the whole network, I would suggest<br>
trying just training the added layer.  If your validation accuracy<br>
is then in line with your training accuracy (even if neither is<br>
good), that would tend to suggest that you don’t have enough<br>
training data in comparison with the number of parameters<br>
(that you are training) in your network.</p>
<aside class="quote no-group">
<blockquote>
<ol start="3">
<li>How can I debug my training/validation process? What can go wrong?</li>
</ol>
</blockquote>
</aside>
<p>Well, you could try to double-check predictions for some of<br>
both your training and validation images.  Hopefully you are<br>
using nearly exactly the same code for calculating your<br>
training and validation accuracy.  If you have two copies<br>
of copy-pasted code you’ve opened a a place for a typo<br>
or other error to creep in.</p>
<p>One quick check would be to pump your training set through<br>
your validation-accuracy code and see if you reproduce your<br>
training accuracy.</p>
<p>Good luck.</p>
<p>K. Frank</p>

grobelny: <ol>
<li>For training I have 400 original files times 4 rotations times 2 (flop) which would give about 3200 pictures times random crops (256x256-&gt;224x224 - around 1000?). Since this is random not all of them will be used. For validation I have a separate set of 12 pictures - not much, but I think a good start.</li>
<li>Yes all the rotations are included on purpose.</li>
<li>The training and validation datasets are completely independent.</li>
<li>Yes the numbers look like this is indeed the second class. The problem was that output was very, very similar for all validation images - it didn’t seem to depend on input much.</li>
<li>I train the whole network. I am not sure that training just the last layer would be enough as normally the resnet used for image classification should be rotation invariant - quite opposite of what  want to do here.</li>
<li>Good point about trying to pump training data through validation pipeline. Thanks a lot.</li>
</ol>
<p>While trying to simplify the code for validation I figured out that doing the scaling externally using ImageMagick and removing the scaling code from my program fixes the issue - my validation accuracy is in excess of 80%. Now I need to figure out what was wrong with image resizing using skimage.transform.resize…</p>