Finding small text in image - best method?

username: stevew77
cooked: <p>Hi</p>
<p>I’m new to python and machine vision.</p>
<p>I have a need to find a fairly short string of text ( 2-4 digits ) in a photo of a cows head ( its ID tag ) .</p>
<p>The text may be rotated slightly by up to +/-  20 degrees.</p>
<p>Then I need to extract the text from the photo.</p>
<p>The text could also have a thick black boundary around it, or it may not.</p>
<p>Does anyone have a suggestion as to an approach for this please?</p>
<p>Some people have suggested ROI and HOG, however as I’m not experienced in this I need general guidance initially.</p>
<p>I’m thinking I could maybe identify the shape of a cow, then zoom in on the cow head area so I’m in the general area where the text is, then find the text? How would I do that ( in broad terms? ) or is this a dumb idea?</p>
<p>Any suggestions welcome! I’m enjoying machine vision, however there is much to learn.</p>

username: ptrblck
cooked: <p>I assume you already have the extracted text for the dataset.<br>
It might be a good idea to use some detection model (deep learning or not) to find the text box in the first step and extract the text using these extracted text boxes.<br>
Do you have the coordinates of the text boxes?</p>
<p>Also, could you post an image of a cow with the corresponding text?<br>
I’m curious where the cows have some kind of text on their heads.</p>

username: KFrank
cooked: <p>Hello Steve!</p>
<aside class="quote no-group quote-modified" data-username="stevew77" data-post="1" data-topic="49338" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/a3d4f5/48.png" class="avatar"> stevew77:</div>
<blockquote>
<p>…<br>
I have a need to find a fairly short string of text ( 2-4 digits ) in a photo of a cows head ( its ID tag ) .</p>
<p>The text may be rotated slightly by up to +/-  20 degrees.</p>
<p>Then I need to extract the text from the photo.<br>
…<br>
Does anyone have a suggestion as to an approach for this please?<br>
…</p>
</blockquote>
</aside>
<p>The general task you are asking about is called “object<br>
detection.”  There is a lot of literature about it that discusses<br>
a number of well-developed methods and algorithms.</p>
<p>Let me suggest one standard approach.  It isn’t necessarily<br>
the “best,” but it is conceptually straightforward, and<br>
relatively easy to implement.  This effective, but somewhat<br>
old-school method (that isn’t necessarily the most efficient)<br>
runs as follows:</p>
<p>First build a binary object classifier – that is, is this object<br>
X or not? – and then apply your object classifier to windows<br>
(of various sizes) that you slide across your image.</p>
<p>In more detail:</p>
<p>Manually annotate your training data.  That is, make a<br>
set of scaled, centered, cropped images that contain<br>
cow ids snipped out of your larger images.  These are<br>
your “positive” – “yes, cow id” training images.  Also<br>
make a similar set of scaled, cropped images snipped<br>
out of your larger images – a cow’s tail, part of a tractor,<br>
a bale of hay.  These are your “negative” – “no cow id”<br>
training images.</p>
<p>These should be smallish images.  Depending on the typical<br>
aspect ratio of a cow id, these might be, say, 32x16 pixel<br>
images.  I doubt that you will need to orient (rotate to the<br>
horizontal) your cow ids – 20 degrees shouldn’t be a problem<br>
for training a classifier.</p>
<p>Now train a binary cow-id classifier.  (Since this is the<br>
pytorch forum, we imagine that you train a simple<br>
convolutional neural network – although a fully-connected<br>
network would likely work fine – but you could also build<br>
a non-neural-network classifier.)</p>
<p>To make your detector, slide windows of various sizes (and<br>
possibly aspect ratios) across your image, and rescale the<br>
contents of your window to the 32x16 size used to train your<br>
classifier.  When a window slides over a cow id, your classifier<br>
should return “yes”, and you have a “hit.”  When the window is<br>
over something else, your classifier should return “no.”</p>
<p>Lastly, as your window slides across a cow id, you are<br>
likely to get several hits for the same cow id.  (Windows<br>
of different sizes are also likely to give you duplicate<br>
hits.)  You want the best hit for any give cow id, so you<br>
apply what is called “non-maximal suppression.”  For<br>
this you want your classifier to provide not just a “yes”<br>
or “no,” but also the “probability” or “confidence”<br>
underlying the “yes” or “no.”  (Let’s say that your<br>
network has a single output – a “logit” – that is fed<br>
into <code>BCEWithLogitsLoss</code> when training your network.<br>
You could use this logit as your confidence.)  For<br>
non-maximal suppression, you look at the confidence<br>
of each of the hits in a cluster of neighboring hits,<br>
and keep only the hit with the largest confidence.</p>
<p>The location of the sliding window, together with its<br>
size (and possibly aspect ratio) for this largest-<br>
confidence hit is the bounding box of your detected<br>
cow id that you can then feed to further processing.</p>
<aside class="quote no-group">
<blockquote>
<p>I’m thinking I could maybe identify the shape of a cow, then zoom in on the cow head area so I’m in the general area where the text is, then find the text? How would I do that ( in broad terms? ) or is this a dumb idea?</p>
</blockquote>
</aside>
<p>This isn’t a dumb idea – you would be using substantive<br>
“semantic” structure of your images, that is, cow ids appear<br>
on cow heads, and not on bales of hay, etc.</p>
<p>I would image that unless text appears in lots of other places<br>
in your images – license plates on tractors, cow id labels<br>
stuck to the sides of barns – just detecting the cow ids<br>
directly will be simpler and easier to implement.</p>
<p>(Even if your images contain a lot of other text, as long as<br>
the cow-id text is somehow distinctive – a certain font, thick<br>
black borders – you could still train just on the cow ids,<br>
provided you make sure to include representative non-cow-id<br>
text in your “not” training samples.)</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: stevew77
cooked: <p>Hi Ptrblk, cows usually have identity tags connected to thier ears that has a 4 digit number of consistent shape and size.</p>
<p>Hi KFrank, thank you for your detailed “how to”. I will now go and research some more and no doubt may have a few questions.  One of the things I find is that as its a big field of study, I’m trying to bolt together existing solutions to make it all work, although I think I will trash and need to rebuild my ubuntu laptop a few times.  <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Would you recommend re-imaging a laptop or maybe working within a virtual machine?</p>
<p>Cheers</p>
<p>Steve</p>