So... What did we learn?

username: shartzog
cooked: <p>If you’d like some background on the ridiculous project I’m working on, see <a href="https://discuss.pytorch.org/t/wasting-my-time/92512">here</a>, <a href="https://discuss.pytorch.org/t/exception-has-occurred-picklingerror-cant-pickle-class-torch-c-variablefunctions-its-not-the-same-object-as-torch-c-variablefunctions/91660">here</a>, and <a href="https://github.com/shartzog/CovidCNN" rel="noopener nofollow ugc">here</a>, but I’ll give the basics below to save you the time…  <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Using BCELoss with a final sigmoid activation, I’ve trained a vanilla CNN to produce a vector of likelihoods for 4 category labels given an input image.  My label targets are length 4 multi hot vectors, e.g. [1, 0, 1, 0].  My input images are produced from tabular data (12 binary columns indicating presence or absence of an attribute for an individual + that individual’s age) by creating a pixel (dtype=float) for every unique three length permutation of column values and then setting the alpha channel for every pixel = age/100.  They look something like this:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/4/743cd4e519db4c3552a1a5b1cc5a0a231a8d5cc4.png" alt="image" data-base62-sha1="gAhIuUqULsXrUWsDMqLI3K8545C" width="516" height="260"></p>
<p>After training, my inference results seem to pass the smell test.  The likelihoods at least seem directionally accurate (e.g. an individual’s risks increase with age and number of preconditions).</p>
<pre><code class="lang-auto">Patient:
36 yo, Female, Tobacco Use
Predicted Label Likelihoods:
['14.41% Hospitalized', '2.05% Intubated', '1.19% Deceased', '10.99% Pneumonia']
Actual Labels:
['none']

Patient:
54 yo, Male, Diabetes, Obesity
Predicted Label Likelihoods:
['59.34% Hospitalized', '14.47% Intubated', '26.84% Deceased', '46.73% Pneumonia']
Actual Labels:
['none']

Patient:
45 yo, Female
Predicted Label Likelihoods:
['20.25% Hospitalized', '3.38% Intubated', '2.51% Deceased', '15.03% Pneumonia']
Actual Labels:
['none']
</code></pre>
<p>With my “smell test” looking promising, I wanted to find a way to visualize model performance across my entire validation set, so I decided to ‘back-predict’ an age distribution for the individuals in each label category using my model and then plot those predicted distributions against the actuals.  There were a number of steps required to create both the predicted and actual distributions, but here are the key details:</p>
<ul>
<li>The predicted count of individuals with a given label for a given age was calculated to be the sum of the predicted likelihoods for that label for each sample with that age in the validation dataset.</li>
<li>The actual count was calculated the same way using the target vector</li>
</ul>
<p>Here’s the result of that effort for two different training runs with the same underlying model structure for two different optimizers, one that used the SGD optimizer with p=0.9 and one that used Adam:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/8/083003bb186c916e1d09f1bc2ef18f7dedd11279.png" data-download-href="https://discuss.pytorch.org/uploads/default/083003bb186c916e1d09f1bc2ef18f7dedd11279" title="image|"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/083003bb186c916e1d09f1bc2ef18f7dedd11279_2_296x500.png" alt="image|" data-base62-sha1="1aqGOkBux6HU3ZWqarh2xNqFHCx" width="296" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/083003bb186c916e1d09f1bc2ef18f7dedd11279_2_296x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/083003bb186c916e1d09f1bc2ef18f7dedd11279_2_444x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/083003bb186c916e1d09f1bc2ef18f7dedd11279_2_592x1000.png 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image|</span><span class="informations">886×1492 139 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
My image squished a bit, but the top 4 plots are for a model trained with Adam, and the bottom 4 are for an equivalent model (everything was the same except the optimizer) that was trained with SGD.  So now (finally :)) my questions:</p>
<ol>
<li>I think it’s clear that the fit of the predicted vs. actual age distributions is telling me <em>something</em>, but what exactly?  I can imagine a model that actually performs terrible during inference that nevertheless produces an exact replica of the actual age distribution.  E.g. the model could predict a likelihood of 0.0001 for every even sample and 0.9999 for every odd sample and the actual labels could be 1 for every even and 0 for every odd.  My distributions would still match, but my predictions would be exactly the opposite of what they should be.  With that in mind, can I make any statements at all about the overall accuracy of my model based on these results?</li>
<li>A match that’s <em>too</em> good might actually indicate overfitting rather than superior performance.  If the distributions match extremely closely, should I be pleased or worried?  In other words, would an ideal result be an exact match or would some slight deviations be preferable?</li>
<li>Given my odd use case, I came to the conclusion that it would be counterproductive to apply any transforms to my training images.  Should I still consider transforms (e.g. rotations, crops, zooming in/out, etc.) even though my images are just “calculated” directly from concrete raw data?  Should I think about overfitting differently as well?</li>
<li>Adam appears to be a far superior optimizer for my use case.  Is that just <em>generally</em> true (i.e. that Adam outperforms SGD) or is it possible to look for certain underlying conditions in a dataset to determine ahead of time that Adam would be the better choice?</li>
<li>Does anyone have suggestions for existing libraries that might have useful validation tools given my oddball use case?</li>
</ol>
<p>Thoughts appreciated…</p>

username: shartzog
cooked: <p>I’ve generated a few more plots, but I’m still struggling to interpret what it all might mean…  Ideas welcome!  <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/d/5da5f75ae90f533fcd0b81ea71769cfdb5fdf679.png" data-download-href="https://discuss.pytorch.org/uploads/default/5da5f75ae90f533fcd0b81ea71769cfdb5fdf679" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5da5f75ae90f533fcd0b81ea71769cfdb5fdf679_2_690x220.png" alt="image" data-base62-sha1="dmrZ7gANVnFnOznIygHgv8YQJbX" width="690" height="220" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5da5f75ae90f533fcd0b81ea71769cfdb5fdf679_2_690x220.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5da5f75ae90f533fcd0b81ea71769cfdb5fdf679_2_1035x330.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/d/5da5f75ae90f533fcd0b81ea71769cfdb5fdf679.png 2x" data-dominant-color="B6A1AE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1254×401 75.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Created with methods similar to those described for creating the age distribution.  Look at the upper left corner of the left and middle map.  The left map shows that 67% of hospitalized patients in the validation set were Male.  The middle map shows that the model would have predicted that 66% of hospitalized patients would have been Male given the data in the validation set.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/a/7ab6ca8d030eb059393fdad34db33103c3942133.png" data-download-href="https://discuss.pytorch.org/uploads/default/7ab6ca8d030eb059393fdad34db33103c3942133" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7ab6ca8d030eb059393fdad34db33103c3942133_2_624x500.png" alt="image" data-base62-sha1="hvzSxOCT9hOfr7vakQiaGIwRp1V" width="624" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7ab6ca8d030eb059393fdad34db33103c3942133_2_624x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7ab6ca8d030eb059393fdad34db33103c3942133_2_936x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/a/7ab6ca8d030eb059393fdad34db33103c3942133.png 2x" data-dominant-color="F7F5F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1001×802 171 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
This plot was my attempt to generalize my predictions to show “risk curves” according to age given a specific set of preconditions.  I can reproduce this to compare any set of attribute (aka precodition) combinations.  EDIT: crosses represent aggregate actual data.  The y error bar is standard error and the x is the age range considered.</p>