Best way to preprocess image

username: MichaelMMeskhi
cooked: <p>Hello,</p>
<p>I am trying to decide what is the best way to preprocess an image that I have. Basically, imagine a photo of a table that has 10 rows and 3 columns. Each cell contains a handwritten digit. I built a MNIST model and retrained with my custom data. But my question is, what is the best way to pass in such input? I want to classify each row numbers for example:</p>
<hr>
<h2>| 5 | 6 | 0 |</h2>
<h2>| 0 | 1 | 2 |</h2>
<p>Each row is a 3 digit number. I was thinking of since image size is fixed, divide into each row then each row into a single digit and pass into model for inference then reconstruct results. I looked into using a pretrained SVHN model but the data differs from handwritten digits.</p>
<p>Thank you for any feedback!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="MichaelMMeskhi" data-post="1" data-topic="91418">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michaelmmeskhi/48/52095_2.png" class="avatar"> MichaelMMeskhi:</div>
<blockquote>
<p>I was thinking of since image size is fixed, divide into each row then each row into a single digit and pass into model for inference then reconstruct results.</p>
</blockquote>
</aside>
<p>This sounds like a reasonable suggestion. Especially, if your input images follow a specific pattern and you could easily isolate each digit.</p>
<p>What kind of model did you look at and how is the input different?</p>

username: MichaelMMeskhi
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I appreciate you taking the time to reply to me! The model I looked into was trained on SVHN dataset (street view house numbers). It can do up to 5 digit recognition in one go but the problem with that is the data used has bounding boxes configured and they are not handwritten digits. If I were to build something like that, I would have to do lots of data preprocessing. I thought going with a good MNIST model would be far easier. Only thing needed was how to process the image as I mentioned.</p>
<p>Thank you again,</p>

username: ptrblck
cooked: <p>Thanks for the information.<br>
I would try to avoid the preprocessing with bounding boxes, as this would give you more flexibility but would also add more complexity to the code, which is apparently not needed for your use case.<br>
Since the digits are located in the same zones, you could write a <code>slice_image</code> method, which could use passed size to return a batch of digits from a single image.</p>
<p>While there are many ways to write this code, here is a small example using <code>unfold</code>:</p>
<pre><code class="lang-python"># create input tensor
x = torch.stack([torch.ones(32, 32) * idx for idx  in range(16)])
x = x.view(4, 4, 32, 32).permute(0, 2, 1, 3).contiguous().view(4*32, 4*32)
plt.imshow(x.numpy())


def slice_image(image, size):
    batch = image.unfold(0, size, size).unfold(1, size, size) # [4, 4, 32, 32]
    # flatten to [16, 32, 32]
    batch = batch.contiguous().view(-1, size, size)
    # add channel dim
    batch = batch.unsqueeze(1)
    return batch
</code></pre>
<p>Note that you would create a batch of 16 samples in this way, so the <code>batch_size</code> in the <code>DataLoader</code> would be a multiplicative factor and you would need to flatten the batch dimensions in your training loop:</p>
<pre><code class="lang-python">for data in loader:
    # data.shape is [batch_size, 16, 1, 32, 32]
    data = data.view(-1, 1, 32, 32)
    # data.shape is [batch_size*16, 1, 32, 32]
</code></pre>

username: MichaelMMeskhi
cooked: <p>I am sorry, but correct me if I understood this correctly. You created a random “image” and then batch each square as a image ready to load for inference?</p>

username: ptrblck
cooked: <p>I created a fake image input, which contains 16 smaller blocks, each in the shape <code>32x32</code>.<br>
This would correspond to the input you are already dealing with.<br>
The <code>slice_image</code> method uses <code>unfold</code> to  create a tensor of <code>[16, 1, 32, 32]</code> which can be called e.g. in <code>Dataset.__getitem__</code>.<br>
Since the <code>Dataset</code> already returns a batch of images (of the “small” <code>32x32</code> images), you would have to reshape the final <code>data</code> tensor, if you are using a <code>DataLoader</code>, since it would add another batch dimension to it.</p>