Saving Loading problem

dneprdva: <p>Hi guys, I have a problem with loading model.<br>
I am training model with batch size 32 of training DataLoader, then make predictions with batch size 1 of testing DataLoader and everything works fine. But if I save this model and then load</p>
<blockquote>
<p>model_loaded.load_state_dict(torch.load(f=MODEL_SAVE_PATH))</p>
</blockquote>
<p>it doesn’t allow me to predict anymore. There is such mistake</p>
<blockquote>
<p>ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 80])</p>
</blockquote>
<p>I can’t understand what has been loaded wrong.</p>

srishti-git1110: <aside class="quote no-group" data-username="dneprdva" data-post="1" data-topic="162578">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dneprdva/48/51640_2.png" class="avatar"> dneprdva:</div>
<blockquote>
<p>it doesn’t allow me to predict anymore</p>
</blockquote>
</aside>
<p>Could you please confirm whether or not you are using <code>model.eval()</code> while inferencing after loading the model?</p>

dneprdva: <p>No, it looks like this</p>
<blockquote>
<p>y_pred_list = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
with torch.inference_mode():<br>
for X_batch in test_loader:<br>
X_batch = X_batch.to(device)<br>
y_test_pred = model_loaded(X_batch)<br>
_, y_pred_tags = torch.max(y_test_pred, dim = 1)<br>
y_pred_list.append(y_pred_tags.cpu().numpy())</p>
</blockquote>

srishti-git1110: <p>Could you please try putting <code>model_loaded.eval()</code> outside the inference_mode context manager and post here if the error persists?</p>

dneprdva: <p>You gave me hint, thank you so much. I already did it and there is no mistake.<br>
There is another problem, the data is very big and inference of this code takes too much time.<br>
Do you have any idea how to speed up this code? I think the problem is in cycling and appending every batch. Thanks in advance.</p>

srishti-git1110: <aside class="quote no-group" data-username="dneprdva" data-post="5" data-topic="162578">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dneprdva/48/51640_2.png" class="avatar"> dneprdva:</div>
<blockquote>
<p>I already did it and there is no mistake.</p>
</blockquote>
</aside>
<p>Do you mean the error in the first post is gone with <code>eval()</code>?</p>
<aside class="quote no-group" data-username="dneprdva" data-post="5" data-topic="162578">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dneprdva/48/51640_2.png" class="avatar"> dneprdva:</div>
<blockquote>
<p>Do you have any idea how to speed up this code?</p>
</blockquote>
</aside>
<p>I would think using multiple GPUs could help generally, but I personally cannot help on this part, sorry.<br>
Since, you are already doing it in inference mode, you don’t have to worry about the expensive gradient computations.</p>
<aside class="quote no-group" data-username="dneprdva" data-post="5" data-topic="162578">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dneprdva/48/51640_2.png" class="avatar"> dneprdva:</div>
<blockquote>
<p>I think the problem is in cycling and appending every batch.</p>
</blockquote>
</aside>
<p>Didn’t quite get you here, could you please elaborate?</p>

dneprdva: <ol>
<li>Yes, with model.eval() everything started to work well.</li>
<li>It is idea, I will check.</li>
<li>Well, as I understand this code, it takes every batch, makes prediction (in my case inference is a tensor of 4 digits), then takes max out of these 4 digits and append result to list. So, I was thinking is there any idea to optimize this cycle?</li>
</ol>

srishti-git1110: <aside class="quote no-group" data-username="dneprdva" data-post="7" data-topic="162578">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dneprdva/48/51640_2.png" class="avatar"> dneprdva:</div>
<blockquote>
<p>it takes every batch, makes prediction (in my case inference is a tensor of 4 digits), then takes max out of these 4 digits and append result to list. So, I was thinking is there any idea to optimize this cycle?</p>
</blockquote>
</aside>
<p>Please post an executable snippet of the part you’d like potential help on. I can then try to if there’s a scope for optimisation.</p>

dneprdva: <p>I don’t really know how to post executable snippet. <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face_with_tear.png?v=12" title=":smiling_face_with_tear:" class="emoji" alt=":smiling_face_with_tear:" loading="lazy" width="20" height="20"><br>
Post the whole model?<br>
Don’t worry, it is ok. I can wait for inference.</p>

ptrblck: <p>You are synchronizing your code by moving the data to the CPU and transforming it into a numpy array:</p>
<pre><code class="lang-python">y_pred_list.append(y_pred_tags.cpu().numpy())
</code></pre>
<p>If you want to avoid this sync, store the detached CUDATensors directly in the <code>list</code> assuming you have enough GPU memory:</p>
<pre><code class="lang-python">y_pred_list.append(y_pred_tags.detach())
</code></pre>
<p>The <code>.detach()</code> call might not be needed if this tensor was already created in a <code>no_grad()</code> block, but it also doesn’t hurt to be explicit here.</p>