Using text within an image to aid in image classifcation

username: melampus123
cooked: <p>I am using a pre-trained resnet50 model to do image classification on common candies seen in American grocery stores. I have 103 classes and 1,400 sample images. The model works relatively well however its most common mistake is confusing different types of specific candies (ex: sour patch kids vs. sour patch kids or Hersheys dark chocolate with milk chocolate). However, the text on the packaging is very different.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/9/79089ab35feabc100a8c7fe0bec9a471942d22c6.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/79089ab35feabc100a8c7fe0bec9a471942d22c6" title="Screen Shot 2022-11-26 at 5.50.36 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/79089ab35feabc100a8c7fe0bec9a471942d22c6_2_424x500.jpeg" alt="Screen Shot 2022-11-26 at 5.50.36 PM" data-base62-sha1="hgIcVFyQr3p5i68K5hpMPMs51CS" width="424" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/79089ab35feabc100a8c7fe0bec9a471942d22c6_2_424x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/79089ab35feabc100a8c7fe0bec9a471942d22c6_2_636x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/79089ab35feabc100a8c7fe0bec9a471942d22c6_2_848x1000.jpeg 2x" data-dominant-color="B6B163"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-11-26 at 5.50.36 PM</span><span class="informations">1066×1256 263 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/f/6ffe0d868279fe8530a32fb8c5907f8977242bdd.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/6ffe0d868279fe8530a32fb8c5907f8977242bdd" title="Screen Shot 2022-11-26 at 5.50.31 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6ffe0d868279fe8530a32fb8c5907f8977242bdd_2_406x500.jpeg" alt="Screen Shot 2022-11-26 at 5.50.31 PM" data-base62-sha1="fYJiU3zKW6ZmglknvbuShaVBI7j" width="406" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6ffe0d868279fe8530a32fb8c5907f8977242bdd_2_406x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6ffe0d868279fe8530a32fb8c5907f8977242bdd_2_609x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6ffe0d868279fe8530a32fb8c5907f8977242bdd_2_812x1000.jpeg 2x" data-dominant-color="BF9C6B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-11-26 at 5.50.31 PM</span><span class="informations">1178×1450 424 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am wondering if it’s possible to explicitly include some level of OCR (optical character recognition) into the model - or do some sort of ensembling with a torchtext model? I was thinking that when I process the image to also run it through either tesseract or paddleOCR to get the image’s readable text and feed that into a torchtext model? Is there some common paradigm which can be followed when doing image classification where the text within the image can be used to aid in classification? The actual training images I’m using are from within stores so text isn’t always available to aid in the classifcation</p>