Synthetic Image Model Too Accurate?

jmartinpizarro: <p>I’m doing a project for identifying grids with either “X”, “G”, “O” or “ “ letters in it. During the first stage of this research, I am doing a simple CNN to identify the letters that may appear in each cell.</p>
<p>I obtain a val_acc = 99.7% and a test_val of = 99.7% aprox too.</p>
<p>For the dataset, I’m using a synthetic dataset I’ve created that generates letters using different fonts (up to 7 No-Tofu fonts). The dataset has over 4000 grids and, when preprocessed, it has 190000 cells. All images are generated with blur, rotation, etc etc etc</p>
<p><strong>My question here is: Is the model overfitted or is just good at it?</strong></p>
<p>Although I preprocess my dataset and then I split it into train, val and test (I know this is not recommended), I just transform into tensors and normalise (and no labels are used here) so theoricatelly <strong>no data leakage should be ocurring here.</strong></p>
<p>Knowing this, is the model overfitted/having data leakage/other things?</p>

KFrank: <p>Hi Javier!</p>
<aside class="quote no-group quote-modified" data-username="jmartinpizarro" data-post="1" data-topic="222523">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jmartinpizarro/48/78012_2.png" class="avatar"> jmartinpizarro:</div>
<blockquote>
<p>I obtain a val_acc = 99.7% and a test_val of = 99.7% aprox too.</p>
<p>…</p>
<p><strong>My question here is: Is the model overfitted or is just good at it?</strong></p>
<p>…</p>
<p>is the model overfitted/having data leakage/other things?</p>
</blockquote>
</aside>
<p>If your test dataset is truly independent of your train dataset, then your results are solid.</p>
<p>Without “data leakage,” training on just your train dataset can’t overfit your test dataset.</p>
<p>I haven’t done the specific experiment, but I think classifying four different characters should</p>
<p>be a pretty simple tasks, so I’m not surprised that you achieve a 99+% accuracy with even a</p>
<p>straightforward, relatively modest CNN.</p>
<p>As an aside, if your preprocessing includes normalizing over the entire unsplit dataset,</p>
<p>then you do technically have some “data leakage,” although I would very much doubt that</p>
<p>this would affect your results.  In any event, normalizing your train, validation, and test datasets</p>
<p>separately would be a suitable way to deal with such a possibility.</p>
<p>Best.</p>
<p>K. Frank</p>

mycul: <p>^ I second this… You’ve described a pretty easy task. So long as you aren’t testing on the data that you used to train the model, you’re results should be correct.</p>

Gillian: <p>here are some ideas to test your code:</p>
<ul>
<li>use a 5th class in the test set, that has not been seen while training. the test accuracy should be lower. (that will certainly require some code adaptation).</li>
<li>randomize the test labels, without randomization the test data (basically, you test with wrong labels), the test accuracy should be very low.</li>
<li>same for training. randomize train label without randomization train data (basically, you train on random data), the model should not be able to learn anything.</li>
</ul>