How to speed up inference?

username: zhoulukuan
cooked: <p>Due to the large dataset, I have spend lots of time testing. How to speed up this process?<br>
Main code:</p>
<pre><code class="lang-auto"># Some other function such as get args

def validate(valloader, model):
    scores = []
    with torch.no_grad():
        for i_iter, batch in enumerate(valloader):
            images, labels = batch
            images = images.cuda(non_blocking=True)

            preds = model(images)
            outputs = torch.nn.functional.softmax(preds)[:, -1]
            scores.extend(outputs.cpu().numpy())
    return scores

if __name__ == '__main__':
    args = parse_args()
    print("Calling with args:")
    print(args)

    os.environ["CUDA_VISIBLE_DEVICES"] = args.gpus
    cudnn.enabled = True
    cudnn.benchmark = True
    torch.backends.cudnn.deterministic = False
    torch.backends.cudnn.enabled = True

    # Define dataset
    rdataset = MyDataSet(dataset='test', size=224, mode='both')
    pdataset = MyDataSet(dataset='val', size=224, mode='both')
    rloader = data.DataLoader(rdataset, batch_size=args.batch_size, shuffle=False,
                                  num_workers=1)  # , pin_memory=True)
    ploader = data.DataLoader(pdataset, batch_size=args.batch_size, shuffle=False,
                                num_workers=1)  # , pin_memory=True)

    # Define model
    num_classes = 2
    model = resnet50(pretrained=False, num_classes=num_classes)
    model.eval()
    model.cuda()

    prefix = args.model.split('/')
    pattern = prefix[-1]
    prefix = '/'.join(prefix[:-1])

    all_files = os.listdir(prefix)
    all_models = [os.path.join(prefix, f) for f in all_files if pattern in f]

    new_state_dict = model.state_dict().copy()

    for i in range(len(all_models)):
        try:
            old_state_dict = torch.load(all_models[i])
            new_state_dict = dict()
            for k, v in old_state_dict.items():
                new_state_dict[k[7:]] = v
            model.load_state_dict(new_state_dict)
        except:
            print("Fail when loading model %s" % all_models[i])
            continue

        rscores = validate(rloader, model)
        pscores = validate(ploader, model)

        # Other subprocess
</code></pre>

username: zhoulukuan
cooked: <p>I got the cost time of each part:</p>
<pre><code class="lang-auto">dataloder of next batch:
about 20s

inference:
about 0.7s

copy tensor to cpu and get results:
about 0.0001s
</code></pre>
<p>I seems that dataloader cost too much time. How to improve it?</p>

username: Usman_Ali
cooked: <aside class="quote no-group quote-post-not-found" data-username="zhoulukuan" data-post="2" data-topic="53349">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhoulukuan/48/12368_2.png" class="avatar"> zhoulukuan:</div>
<blockquote>
<p>flagged</p>
</blockquote>
</aside>
<p>Try increasing the value of <code>num_workers</code> in Dataloader (e.g. <code>num_workers=8</code>). This way, more sub-processes will be utilized for loading data, hence decreasing data load time.</p>

username: zhoulukuan
cooked: <p>Thank you for your advice. I want to know how to find a suitable num_workers? It has some relationship with the count of cpu?</p>

username: Usman_Ali
cooked: <p>Ideally, a suitable value for <code>num_workers</code> is the minimum value which will give batch loading time &lt;= inference time. This way, when our model is working on inference of previous batch, data-loader would be able to finish reading the next batch in the mean time.</p>
<p>However, the maximum number of <code>num_workers</code> is also dependent on available cpu resources, so we might not always be able to achieve that ideal number of <code>num_workers</code>.</p>

username: Zhipeng_Fan
cooked: <p>In the latter case, would it be good to just set it to <span class="hashtag">#cores</span> on CPU?</p>