Converting Deeplabv3 for inference

Kishor_G: <p>Hi,</p>
<p>I have finetuned DeepLabv3 model for a custom dataset.</p>
<p>I converted the trained model to a jit model using torch.jit.script() and the model worked fine and gave the desired output during inference. But when I try to convert the same model again to jit, the results are undesirable during inference and looks completely poles apart.</p>
<p>I am new to jit, so trying to figure out things.</p>
<p>The model was trained on 2 GPUs, then convert to a single model GPU, then tried converting it to jit.</p>

ptrblck: <aside class="quote no-group" data-username="Kishor_G" data-post="1" data-topic="190808">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kishor_g/48/58165_2.png" class="avatar"> Kishor_G:</div>
<blockquote>
<p>But when I try to convert the same model again to jit, the results are undesirable</p>
</blockquote>
</aside>
<p>Could you explain how you are concerting the model again? Are you calling <code>torch.jit.script</code> on the already scripted model?</p>

Kishor_G: <p>I am calling the torch.jit.script on a loaded state dict, one time was almost a month ago and once today on the same pretrained model, I printed the weights of model they to vary</p>

Kishor_G: <p>I am not calling torch.jit.script on a scripted model</p>

ptrblck: <aside class="quote no-group" data-username="Kishor_G" data-post="3" data-topic="190808">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kishor_g/48/58165_2.png" class="avatar"> Kishor_G:</div>
<blockquote>
<p>I printed the weights of model they to vary</p>
</blockquote>
</aside>
<p>In that case the <code>state_dict</code> seems to have changed and you should check it.</p>

Kishor_G: <p>I am converting from a multigpu (nn.DataParallel) model to single gpu model by this method:</p>
<p>for key in deeplab_model:<br>
new_key = key.replace(‘module.’, ‘’)<br>
new_state_dict[new_key] = deeplab_model[new_key]</p>
<p>Is this right way to do?</p>

ptrblck: <p>It looks correct and <code>model.load_state_dict</code> will also fail if you have mismatching keys.</p>

Kishor_G: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/2/02fbf6636f245cc891497e1da41735f564c1ef46.png" data-download-href="https://discuss.pytorch.org/uploads/default/02fbf6636f245cc891497e1da41735f564c1ef46" title="driveable_2D"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/02fbf6636f245cc891497e1da41735f564c1ef46_2_690x388.png" alt="driveable_2D" data-base62-sha1="qoMu30o9HvzW1E8XgufqgjTPVQ" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/02fbf6636f245cc891497e1da41735f564c1ef46_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/02fbf6636f245cc891497e1da41735f564c1ef46_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/02fbf6636f245cc891497e1da41735f564c1ef46_2_1380x776.png 2x" data-dominant-color="5E5E5E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">driveable_2D</span><span class="informations">1920×1080 6.02 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
This is the inference on the multigpu model</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b84f61b2390acb3fe46877a62289358b39cabc20.png" data-download-href="https://discuss.pytorch.org/uploads/default/b84f61b2390acb3fe46877a62289358b39cabc20" title="driveable_2D"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b84f61b2390acb3fe46877a62289358b39cabc20_2_690x388.png" alt="driveable_2D" data-base62-sha1="qitVAJA8Dun0gHSrTAXq7ePa6ty" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b84f61b2390acb3fe46877a62289358b39cabc20_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b84f61b2390acb3fe46877a62289358b39cabc20_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b84f61b2390acb3fe46877a62289358b39cabc20_2_1380x776.png 2x" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">driveable_2D</span><span class="informations">1920×1080 9.86 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
This is the output on the single GPU model</p>
<p>Both the outputs are before converting to a jit model. So, I think converting from multi gpu to single gpu is causing the issue. Is there a way to convert a multigpu model to single gpu other than the above mentioned way?</p>

Kishor_G: <p>Found this post.</p><aside class="quote quote-modified" data-post="1" data-topic="22838">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/ad7895/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-could-i-train-on-multi-gpu-and-infer-with-single-gpu/22838">How could I train on multi-gpu and infer with single gpu</a> 
  </div>
  <blockquote>
    I train a model on two gpus, and then save the model like this: 
    net = nn.DataParallel(net)
    .....
    torch.save(net, save_path)

Then I load the model and run inference with single gpu: 
   CUDA_VISIBLE_DEIVCES=0 python infer.py

the infer.py is like this: 
    im = cv2.imread('./cropped.jpg')
    im = cv2.resize(im, (224, 224)).transpose(2, 0, 1)
    im = torch.tensor([im, im, im], dtype = torch.float32)
    model_path = './res/model.pytorch'
    model = torch.load(model_path)
    mode…
  </blockquote>
</aside>

<p>Will try this method, and let you know if it works</p>

Kishor_G: <p>Using the above mentioned way to convert nn.Dataparallel to single gpu model works consistently compared to the key replacement in the state_dict. Now I am able to get desired solution and indeed the issue is with the state_dict as foreseen by <a class="mention" href="/u/ptrblck">@ptrblck</a></p>