How to load images for inference in batch

codegypsy1: <p>Hey there I am new to PyTorch. I have a inference code that predicts and classify images. I can predict and classify images one by one, can anyone please help me to classify all the images of a folder in a batch.</p>
<p>Directory structure:<br>
images<br>
… img1.jpg<br>
… img2.jpg<br>
… img3.jpg</p>
<p>Output:<br>
elephant<br>
lion<br>
tiger</p>
<p>How can I load all the image in the folder and predict one by one.<br>
I am using the prediction code as follows:</p>
<pre><code class="lang-auto">def predict_image(image_path):
    print("prediciton in progress")
    image = Image.open(image_path)

    transformation = transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ])

    image_tensor = transformation(image).float()
    image_tensor = image_tensor.unsqueeze_(0)

    if cuda:
        image_tensor.cuda()

    input = Variable(image_tensor)
    output = model(input)

    index = output.data.numpy().argmax()
    return index
</code></pre>
<p>This works for single images, if called again and again the time of execution will increase. Also I am doing inference on a CPU machine.</p>

Amrit_Das: <p>As per my understanding, I wrote this piece of code. To load images and predict.</p>
<pre><code class="lang-auto">data_transforms = {
    'predict': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ])
    }

dataset = {'predict' : datasets.ImageFolder("./data", data_transforms['predict'])}
dataloader = {'predict': torch.utils.data.DataLoader(dataset['predict'], batch_size = 1, shuffle=False, num_workers=4)}

outputs = list()
since = time.time()
for inputs, labels in dataloader['predict']:
    inputs = inputs.to(device)
    output = model(inputs)
    output = output.to(device)
    index = output.data.numpy().argmax()
    print index
</code></pre>
<p>I hope this helps you and solves your problem <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=6" title=":smile:" class="emoji" alt=":smile:"></p>

codegypsy1: <p>Thank You <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"> worked like a charm</p>

Ben_Bowles: <p>What is a device? <code>torch.cuda.current_device()</code> ?</p>

Manuel_Alejandro_Dia: <p>Usually a device is where you do your computations, for example if you use a GPU your device will be something like <code>cuda:0</code> or just <code>0</code>.</p>
<p>So what <code>torch.cuda.current_device()</code> does is to return the identifier of which GPU is currently being used. This can be really helpful for systems with multiple GPUs.</p>
<p><a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.current_device" rel="nofollow noopener">Docs for more information.</a></p>

adijindal30: <p>Does inferencing on the batch reduce the per image inference time?</p>

Manuel_Alejandro_Dia: <p>I guess the answer would depend on your network and input, but according to some tests I ran, time is the same for batch or single image.<br>
<strong>So yes, it reduces the per-image inference time</strong>.</p>
<pre><code class="lang-auto">import torch

single_image = torch.rand(1,3,300,300).cuda()
multiple_images = torch.rand(8,3,300,300).cuda()

network = torch.hub.load('pytorch/vision:v0.9.0', 'resnet50').cuda().eval()

with torch.no_grad():
    %timeit network(single_image)
       #33.2 ms ± 123 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
    %timeit network(multiple_images)
       #33.2 ms ± 399 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
</code></pre>
<p>Running multiple image inference depends on your application and hardware. For example:</p>
<ul>
<li>If you are training, the bigger the batch size the better. Because it will allow you to have a better gradient update, and in the end a faster training time. Remember that inference is not all the computation you perform when executing your model. Things like pre and post-processing also take time, and if you can do these steps at the same time for many images your computation time will be reduced!</li>
<li>If you are using a live system, you don’t want to wait and have many images to do inference in all of them at once. Usually, you want to do inference on the latest available image (batch of 1). Even though the computation time per image is reduced, it may be too slow the overhead caused by the rest of the system, making single image inference the best option.</li>
</ul>

adijindal30: <p>yes you are right and I guess the difference in inference time is quite large when I just using CPU otherwise in the case of GPU, I guess only a little difference in inference time when I did the batch inference. Here is the graph for Resnet-18 inference using GPU, on 256 images.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/5/c5a295b988a8c119a8a391eed4c9964156521e2d.png" alt="download (1)" data-base62-sha1="scmoGrL6bDjzxuzr818k8PnA8pf" width="376" height="262"></p>

sal_zak: <p>how to turn list of images to a batch to do inference per batch?</p>

Manuel_Alejandro_Dia: <p>What do you mean with list of images?<br>
Can you give an example of your code?</p>
<p>Your question is too vague</p>

sal_zak: <p>a python list of numpy array each array represent an image and i fed the list to this function and i want to return the array of predictions<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c753d1b1d976bab8944a3aecd02eccaa2b51908.png" alt="Screenshot from 2023-05-11 14-41-01" data-base62-sha1="mk5FqjqQV5El9Y4eWdDDG4gOhAs" width="313" height="313"></p>

tiramisuNcustard: <p>Since you are working with images, it might be easier to work with ImageFolder. Take a look at its documentation here:</p>
<p><a href="https://pytorch.org/vision/main/generated/torchvision.datasets.ImageFolder.html" rel="noopener nofollow ugc">https://pytorch.org/vision/main/generated/torchvision.datasets.ImageFolder.html</a></p>