What model architecture should I use to extract all text from any image (OCR use case)?

username: vani_anandan
cooked: <p>I am working on a use case where I want to extract all visible text from any kind of image — including medicine packages, product cartons, documents, and natural scenes.<br>
What I want to build:<br>
A deep learning-based OCR model that takes an image as input and returns all the text written on it.<br>
I have tried:</p>
<ol>
<li>CRNN Model (CNN + BiLSTM + CTC loss):<br>
I implemented a basic CRNN using Keras. The model includes:
<ul>
<li>CNN layers for feature extraction</li>
<li>BiLSTM layers for sequence modeling</li>
<li>Dense + CTC loss for character decoding</li>
</ul>
</li>
</ol>
<p>Any architecture suggestions, research paper links, or open-source code would be helpful</p>

username: twoudsma
cooked: <p>Hi, have you tried any VLM-models for OCR? There are many new architectures and strategies that might work for your case! Especially because it is on all kinds of different scenes on which VLMs should also have better capabilities. Some place to start for instance: <a href="https://huggingface.co/collections/Cuiunbo/vlm-for-ocr-664f92901739c75611e35300" class="inline-onebox" rel="noopener nofollow ugc">VLM For OCR - a Cuiunbo Collection</a>.</p>
<p>Another option is to use <a href="https://github.com/tesseract-ocr/tesseract" rel="noopener nofollow ugc">Tesseract</a>, a library that has been around for quite some time, but still has active development. They also have a python wrapper that you can use: <a href="https://pypi.org/project/pytesseract/" class="inline-onebox" rel="noopener nofollow ugc">pytesseract·PyPI</a>.</p>
<p>Finally, you could also find datasets and pretrained detection model for all existing characters and fine tune this model yourself on your data. This is definitely the most work, but also gives you the most freedom to change the architecture and data in any way you want.</p>
<p>Let me know if you need any more information!</p>

username: Hamza_Javaid
cooked: <aside class="quote no-group" data-username="vani_anandan" data-post="1" data-topic="221051" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vani_anandan/48/77477_2.png" class="avatar"> vani_anandan:</div>
<blockquote>
<p>I am working on a use case where I want to extract all visible text from any kind of image — including medicine packages, product cartons, documents, and natural scenes.<br>
What I want to build:<br>
A deep learning-based OCR model that takes an image as input and returns all the text written on it.<br>
I have tried:</p>
<ol>
<li>CRNN Model (CNN + BiLSTM + CTC loss):<br>
I implemented a basic CRNN using Keras. The model includes:</li>
</ol>
<ul>
<li>CNN layers for feature extraction</li>
<li>BiLSTM layers for sequence modeling</li>
<li>Dense + CTC loss for character decoding</li>
</ul>
<p>Any architecture suggestions, research paper links, or open-source code would be helpful</p>
</blockquote>
</aside>
<p>Your current CRNN model has a fundamental limitation, it’s designed to read text that flows in a single, predictable direction, like reading a single line of text. This works for simple cases but fails when confronted with the complex scenes where text appears at different angles, sizes, and locations throughout the image.</p>
<p>PaddleOCR in my personal experience is more robust to these noises.</p>
<p>The pre-trained PaddleOCR models already understand various text styles, but fine-tuning them on your specific use cases - medicine packages and product cartons - will significantly improve accuracy.</p>