Cpu at 100% while GPU remains at 1% on transforms

Jaume_F: <p>Here is the code that I isolate to reproduce the problem.<br>
When doing the inference, the Cpu has 100% load, and the Gpu remains with almost no load.<br>
This is for the inference, so I provide an image that is not heavy, about 400x400px. The model has also a very low wait. Then, the consuming load is done here : image_transformed = transform(image).unsqueeze(0).to(device)<br>
I can’t provide batches or so, so it’s a new image inspected every time… How to reduce the load of the cpu? The inference for the classifier takes only 5ms, but with the cpu at 100%, and my applications runs out of cycle for the other main tasks because of this.</p>
<pre><code class="lang-auto">device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print("Using device:", device)


def main():

    # Load and preprocess an image for inference
    image = Image.open(image_path)

    ###
    # Define Image transformations
    transform = transforms.Compose(
        [transforms.Resize((224, 224)), #224 for the resNet50
        transforms.ToTensor(),
        transforms.Normalize(
            mean=[0.485, 0.456, 0.406],
            std=[0.229, 0.224, 0.225]
        )
    ])

    trainset = torchvision.datasets.ImageFolder(root=PathTrain, transform=transform)
    classes = trainset.classes

    model = torchvision.models.squeezenet1_1(weights=True)
    model.classifier[1] = nn.Conv2d(512, len(classes), kernel_size=(1, 1))
    model.num_classes = len(classes)

    model.load_state_dict(torch.load(PathModel))
    model.eval()

    model.to(device)
    ###

    while True:
            #
            try:
                start_time = time.time()

                # Process image
                image_transformed = transform(image).unsqueeze(0).to(device)

                with torch.no_grad():
                    output = model(image_transformed)
                
                _, indice = torch.max(output, 1)

                elapsed_time = (time.time() - start_time) * 1000
            
            except Exception as e:
                response = 'ERROR'

            finally:
                time.sleep(0.1) 


if __name__ == "__main__":
    main()
</code></pre>

ptrblck: <p>You could try to use CUDA Graphs to reduce the CPU workload as it seems it might be the bottleneck in your application.</p>

Jaume_F: <p>Thanks for the tip. I am reading about it, so it’s unknown for me. If you have an example or link, I will appreciate that.</p>

ptrblck: <p>You can check the docs <a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-graphs">here</a> which explain how to manually use it (or via a util. function), but the easier approach might be to use <code>torch.compile</code> with <code>mode="reduce-overhead"</code> which should apply it for you (assuming your model is compatible with CUDA Graphs).</p>

Jaume_F: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="196037">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>torch.compile</code> with <code>mode="reduce-overhead"</code></p>
</blockquote>
</aside>
<p>Isn’t compile not supported in Windows? RuntimeError: Windows not yet supported for torch.compile</p>

ptrblck: <p>Yes, I don’t think <code>torch.compile</code> is supported on Windows due to the lack of OpenAI/Triton support on Windows.</p>

Jaume_F: <p>Mm so I am using Windows. I would like to understand why a bunch of 6 cameras grabbing at 60fps takes only 10% of cpu, and the cpu-&gt;gpu at just 30 images/s takes the 100% cpu, even multithreading or any parallelization task. So, Is not any other method to accelerate/reduce cpu load? Should this values of 100% cpu and 1% gpu accepted as ‘normal’ for inference in production?</p>