F.interpolate and Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

username: max4
cooked: <p>When running model on server I get the following error:</p>
<p>I use</p>
<p><code>F.interpolate(g, size=input_size[2:], mode=upsample_mode)</code> in the forward method<br>
and get following error when running script on the server:</p>
<h3><a name="file-usrlocallibpython310dist-packagestorch_decompdecompositionspy-line-2821-in-upsample_bilinear2d-x_ceil-torchceilxclampmaxin_h-1totorchint64-1" class="anchor" href="#file-usrlocallibpython310dist-packagestorch_decompdecompositionspy-line-2821-in-upsample_bilinear2d-x_ceil-torchceilxclampmaxin_h-1totorchint64-1"></a>File “/usr/local/lib/python3.10/dist-packages/torch/_decomp/decompositions.py”, line 2821, in upsample_bilinear2d x_ceil = torch.ceil(x).clamp(max=in_h - 1).to(torch.int64)</h3>
<h3><a name="runtimeerror-expected-all-tensors-to-be-on-the-same-device-but-found-at-least-two-devices-cuda0-and-cpu-when-checking-argument-for-argument-max-in-method-wrapper_cuda_clamp_tensor-2" class="anchor" href="#runtimeerror-expected-all-tensors-to-be-on-the-same-device-but-found-at-least-two-devices-cuda0-and-cpu-when-checking-argument-for-argument-max-in-method-wrapper_cuda_clamp_tensor-2"></a>RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument max in method wrapper_CUDA_clamp_Tensor)</h3>
<p>When running the same model locally on GPU this error does not appear.</p>
<p>Also this error seems to not appear when running on the server, but directly specifying size, e.g: F.interpolate(g, size=(64,64), mode=upsample_mode), but I need size to be calculated and not specified.</p>
<p>In the following thread (<a href="https://github.com/pytorch/pytorch/issues/76487" class="inline-onebox" rel="noopener nofollow ugc">`F.interpolate` uses incorrect size when `align_corners=True` · Issue #76487 · pytorch/pytorch · GitHub</a>) I have seen suggestion: <a href="https://github.com/pytorch/pytorch/issues/76487#issuecomment-1112653986" class="inline-onebox" rel="noopener nofollow ugc">`F.interpolate` uses incorrect size when `align_corners=True` · Issue #76487 · pytorch/pytorch · GitHub</a></p>
<p>Do you know any reason for this problem or any workaround for this? I do not think that this related to moving model or tensor to cuda.</p>

username: ptrblck
cooked: <p>I don’t know where the device mismatch comes from so could you print all inputs to <code>F.interpolate</code>?</p>

username: max4
cooked: <p>It seems it related to the following incident: <a href="https://github.com/pytorch/pytorch/issues/94174" class="inline-onebox" rel="noopener nofollow ugc">[pt2] The min and max parameters of torch.clamp do not support numpy format · Issue #94174 · pytorch/pytorch · GitHub</a>, somehow related how torch.clamp works.</p>
<p>I am not completely able to follow explanation on what is happening with dynamo and inductor, but when I downgraded PyTorch to 2.0.1 on the server (to be the same as on the local machine, where it worked without this error) - then it worked on server as well.</p>
<p>So basically running the same code in this image at server: <a href="https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel-23-05.html#rel-23-05" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Release 23.05 - NVIDIA Docs</a> or with PyTorch 2.0.1 locally, F.interpolate functions does not throw the device mismatch error.<br>
When running in the following image: <strong>rel-23-08</strong> it throws the above mentioned device mismatch error unless size explicitly fixed with numbers e.g. size=(64,64).<br>
So I assume that there was some change between PyTorch 2.0.1 and PyTorch 2.1, which causes this error.</p>
<p>As to the inputs to F.interpolate:</p>
<ol>
<li>I have tried input_size[2:] - which is torch.Size - works on PyTorch 2.0.1, but not PyTorch 2.1</li>
<li>I have tried [int(h), int(w)], (int(h), int(w)) - works on PyTorch 2.0.1, but not PyTorch 2.1</li>
<li>and (64,64) - the only one which worked on PyTorch 2.1 and PyTorch 2.0.1</li>
</ol>

username: alessandro_bonvini
cooked: <p>Hi,<br>
I’m experiencing a similar issue with version 2.3.0.</p>
<p>When I train my model everything runs without any errors, but when I try to save the trained model with <strong>‘torch.jit.trace’</strong> I get the following error:</p>
<p><strong>RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument max in method wrapper_CUDA_clamp_Tensor)</strong></p>
<p>When I call <strong>‘torch.jit.trace’</strong> both the model and the input are on the GPU.</p>
<p>This is the function of my code that generates the error:</p>
<pre><code class="lang-auto">flow_map = F.interpolate(
                input=-prob,
                size=self.input_size,
                mode="bilinear",
                align_corners=False,
            )
</code></pre>
<p>Where <strong>‘-prob’</strong> is a torch cuda tensor and <strong>‘self.input_size’</strong> is a tuple (336, 336).</p>
<p>The exception is actually raised by the <strong>‘get_values’</strong> function of the <strong>‘torch_decomp\decompositions.py’</strong> module at line 3488:</p>
<pre><code class="lang-auto">xp1 = (x + 1).clamp(max=inp_size - 1)
</code></pre>
<p>It somehow happens that, during scripting, the <strong>inp_size</strong> variable becomes a CPU tensor, while during training the variable is an int.<br>
This seems to be the problem, because if I do this check on <strong>inp_size</strong>:</p>
<pre><code class="lang-auto">        if type(inp_size) is not int:
            inp_size = inp_size.to(x.device)
        xp1 = (x + 1).clamp(max=inp_size - 1)
</code></pre>
<p>The error does not occur anymore.</p>
<p>If I downgrade to version 2.0.1 everything works.</p>
<p>I would be very grateful if you could help me because this issue is preventing us from upgrading the PyTorch version.</p>
<p>Thanks in advance!</p>

username: bayesianroad
cooked: <p>I was not able to fix it, just downgraded Pytorch to 2.0.1.<br>
I assume that this bug (<a href="https://github.com/pytorch/pytorch/issues/94174" class="inline-onebox" rel="noopener nofollow ugc">[pt2] The min and max parameters of torch.clamp do not support numpy format · Issue #94174 · pytorch/pytorch · GitHub</a>) has to be fixed.<br>
But maybe <a class="mention" href="/u/ptrblck">@ptrblck</a> could elaborate.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="alessandro_bonvini" data-post="4" data-topic="196117">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alessandro_bonvini/48/61813_2.png" class="avatar"> alessandro_bonvini:</div>
<blockquote>
<p>I would be very grateful if you could help me because this issue is preventing us from upgrading the PyTorch version.</p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="alessandro_bonvini" data-post="4" data-topic="196117">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alessandro_bonvini/48/61813_2.png" class="avatar"> alessandro_bonvini:</div>
<blockquote>
<p>When I train my model everything runs without any errors, but when I try to save the trained model with <strong>‘torch.jit.trace’</strong> I get the following error:</p>
</blockquote>
</aside>
<p><code>TorchScript</code> is deprecated and in “maintenance” mode which means it won’t receive any major updates or bug fixes. You could try to use <code>torch.compile</code> and check if the issue disappears.</p>
<aside class="quote no-group" data-username="bayesianroad" data-post="5" data-topic="196117">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/ea666f/48.png" class="avatar"> bayesianroad:</div>
<blockquote>
<p>I assume that this bug (<a href="https://github.com/pytorch/pytorch/issues/94174">[pt2] The min and max parameters of torch.clamp do not support numpy format · Issue #94174 · pytorch/pytorch · GitHub</a>) has to be fixed.</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/alessandro_bonvini">@alessandro_bonvini</a> uses <code>torch.jit.trace</code> so the issue is unrelated as it’s pointing towards <code>torch.compile</code> treating numpy arrays as <code>CPUTensor</code>s.</p>

username: alessandro_bonvini
cooked: <p>Thank you for the reply!</p>
<p>I’m using <code>torch.script</code> because I need to run on Windows. Additionally, I have a runtime engine written in C++ using libtorch that reads the scripted model and executes it at runtime.</p>
<p>Is there any other way to do this? I mean training a model in PyTorch, saving it, and executing it at runtime with libtorch without knowing the model implementation?</p>
<p>Thank you again!</p>