Performance drop of trained model in production

evgeniititov: <p>Hello everyone,</p>
<p>I am pretty sure this question has been asked hundreds of times. However, I failed to find an answer to my problem.</p>
<p>Context: I’ve trained the ResNet18 model on 2 classes. In training 2700/3100 images per class, on validation: 680/770. The model trained well, I ended up with the accuracy of around 99% and loss around 0.01 - 0.02. Fine tuning, pretrained.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d706925fd7bafb1b85e36ae35f45c373226d4487.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d706925fd7bafb1b85e36ae35f45c373226d4487" title="Training3"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d706925fd7bafb1b85e36ae35f45c373226d4487_2_690x350.jpeg" alt="Training3" data-base62-sha1="uGcJDBudKILt7Xi8hOD18xL4Nn1" width="690" height="350" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d706925fd7bafb1b85e36ae35f45c373226d4487_2_690x350.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d706925fd7bafb1b85e36ae35f45c373226d4487_2_1035x525.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d706925fd7bafb1b85e36ae35f45c373226d4487_2_1380x700.jpeg 2x" data-dominant-color="F8F9F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Training3</span><span class="informations">1558×791 80 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>However, when we integrated the model in our system we quickly noticed it is half not as good as on validation data even though the data we’ve been feeding to the model is pretty much the same. I am 99% sure the problem is in preprocessing, however I am unable to find where I made a mistake. I’ve been staring in the code for too long, if anyone could have a quick glance please. Probably you could spot what I did wrong.</p>
<p>On step 1, we read N frames of a video (or just 1 photo) and load it to GPU. No preprocessing done. Just original images:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/6/466f0e086e0a63a526c24d0cbad8774e2bcf65b4.png" data-download-href="https://discuss.pytorch.org/uploads/default/466f0e086e0a63a526c24d0cbad8774e2bcf65b4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/466f0e086e0a63a526c24d0cbad8774e2bcf65b4_2_662x500.png" alt="image" data-base62-sha1="a35m7s6sf98CS34vr39wzXXe404" width="662" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/466f0e086e0a63a526c24d0cbad8774e2bcf65b4_2_662x500.png, https://discuss.pytorch.org/uploads/default/original/3X/4/6/466f0e086e0a63a526c24d0cbad8774e2bcf65b4.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/6/466f0e086e0a63a526c24d0cbad8774e2bcf65b4.png 2x" data-dominant-color="2C2E27"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">887×669 83.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Then we have the YOLO net that finds objects for us. Using the coordinates of detected objects, we slice torch tensors on GPU and accumulate them in a list and send to our classification ResNet model. The model will need to do all necessary preprocessing because uploaded to GPU images underwent none.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/7/07e0260eb09448b41cfb2eca07ad7dbfe5b5bb0a.png" data-download-href="https://discuss.pytorch.org/uploads/default/07e0260eb09448b41cfb2eca07ad7dbfe5b5bb0a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/7/07e0260eb09448b41cfb2eca07ad7dbfe5b5bb0a_2_690x494.png" alt="image" data-base62-sha1="17FzS5dDrS819z1927cMjUyCggW" width="690" height="494" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/7/07e0260eb09448b41cfb2eca07ad7dbfe5b5bb0a_2_690x494.png, https://discuss.pytorch.org/uploads/default/original/3X/0/7/07e0260eb09448b41cfb2eca07ad7dbfe5b5bb0a.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/7/07e0260eb09448b41cfb2eca07ad7dbfe5b5bb0a.png 2x" data-dominant-color="2B2C26"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">997×715 82.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The normalization function:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/7/47d08f2b1d3295e16dae324c784dc312258d3ae0.png" data-download-href="https://discuss.pytorch.org/uploads/default/47d08f2b1d3295e16dae324c784dc312258d3ae0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/47d08f2b1d3295e16dae324c784dc312258d3ae0_2_690x128.png" alt="image" data-base62-sha1="afiJFfymClBGdUJnj7U1u4UMK6Q" width="690" height="128" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/47d08f2b1d3295e16dae324c784dc312258d3ae0_2_690x128.png, https://discuss.pytorch.org/uploads/default/original/3X/4/7/47d08f2b1d3295e16dae324c784dc312258d3ae0.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/7/47d08f2b1d3295e16dae324c784dc312258d3ae0.png 2x" data-dominant-color="2C2D28"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">860×160 17.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I do understand that I must have failed to follow the preprocessing I used during training. Which is here:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b9b2f8d653627f95c617b0dc463eab06b2dfb708.png" data-download-href="https://discuss.pytorch.org/uploads/default/b9b2f8d653627f95c617b0dc463eab06b2dfb708" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b9b2f8d653627f95c617b0dc463eab06b2dfb708.png" alt="image" data-base62-sha1="quLMdVRIrwnQPkfyI6RMZJoyUBy" width="690" height="183" data-dominant-color="31332F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">697×185 18 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>If anyone could please have a look and let me know what I did wrong because I cannot find the problem.</p>
<p>I’ve studied what happens with validation images under the hood in Resize(), ToTensor() and Normalize() steps. What I’m doing seems to be identical, however I am getting much more errors than on validation data, which is very very similar to what the model’s been dealing with after integration.</p>
<p>Thank you very much in advance.<br>
Regards,<br>
Eugene</p>

evgeniititov: <p>Fun fact - when I do image preprocessing and make sure my torch tensor is RGB and not BGR (predict function, beginning of the for loop where I do premute), the model works even slightly worse. I’d expect it to work better though because it was training with RGB images. Thanks ColorJitter().</p>

vgsprasad: <p>Hi,</p>
<p>This is my guess. During training process, both dimensions of image are same. Width and height parameters of resize function were set to ‘input_size’. But, during testing, interpolation function was used with ‘size = image_size’. In this case, the smallest dimension of image is set to this value. The bigger dimension will be scaled accordingly to maintain aspect ratio. During training phase, aspect ratio of image is not maintained.</p>
<p>Thanks<br>
VGS</p>

evgeniititov: <p>Hi <a class="mention" href="/u/vgsprasad">@vgsprasad</a>,</p>
<p>Thanks for your reply. Sorry, the screenshot I attached is a bit misleading. The value of image size is actually (224, 224). So, it is the same as during training <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>Regards,<br>
Eugene</p>