Model.to(device) vs. tensor.to(device)

dkleine: <p>I recently ran into <a href="https://stackoverflow.com/a/59560101" rel="noopener nofollow ugc">this discussion</a> referencing the difference in the assignment of <code>my_model.to(device)</code> and <code>my_tensor.to(device)</code>. While currently the model will be moved <em>in-place</em> to the device, tensors must be assigned to the same variable to be moved to the device (<code>my_tensor = my_tensor.to(device)</code>).</p>
<p>I have searched this forum for similar topics explaining this logic. There a lot of discussions on the specific usage of <code>.to(device)</code>, but none that explicitly addresses why PyTorch implements these methods differently for models vs. tensors</p>
<p>From my point-of-view, the issue with this logic is when using <code>my_model.to(device)</code> along with <code>my_tensor.to(device)</code> a device mismatch will occur, without the problem being directly visible. As a developer, you would need to know which objects are tensors or not in the code (especially hard for PyTorch beginners). Also, the logic is inconsistent with other <em>in-place</em> methods, for example <code>.add_()</code>, that contain an underscore <code>_</code> specifically for this reason.</p>
<p>Is there any particular logic or design philosophy behind this decision to have different behaviors for the same method name? For me, it seems counterintuive that <code>.to(device)</code> behaves differently on object types. Would it make sense to make the behaviour of <code>.to(device)</code> consistent for models and tensors, for example making both <code>my_model.to(device)</code> and <code>my_tensor.to(device)</code> performed <em>in-place</em>?</p>

ptrblck: <aside class="quote no-group" data-username="dkleine" data-post="1" data-topic="220160">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dkleine/48/77066_2.png" class="avatar"> dkleine:</div>
<blockquote>
<p>As a developer, you would need to know which objects are tensors or not in the code (especially hard for PyTorch beginners).</p>
</blockquote>
</aside>
<p>To resolve the inconsistency, you could make sure to always assign the return value for <code>to()</code> calls as:</p>
<pre data-code-wrap="python"><code class="lang-python">model = model.to(device)
tensor = tensor.to(device)
</code></pre>
<p>will also work.</p>
<aside class="quote no-group" data-username="dkleine" data-post="1" data-topic="220160">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dkleine/48/77066_2.png" class="avatar"> dkleine:</div>
<blockquote>
<p>Also, the logic is inconsistent with other <em>in-place</em> methods, for example <code>.add_()</code>, that contain an underscore <code>_</code> specifically for this reason.</p>
</blockquote>
</aside>
<p>I see the <code>inplace</code> definition w.r.t. the actual data and memory. I.e. I expect an inplace operation to process the already allocated memory and perform every operation on values stored in this memory. Moving data from one device to another thus cannot be an inplace operation unless you abstract the memory model and use something like managed memory (which will move memory behind your back to the device).</p>
<aside class="quote no-group" data-username="dkleine" data-post="1" data-topic="220160">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dkleine/48/77066_2.png" class="avatar"> dkleine:</div>
<blockquote>
<p>Is there any particular logic or design philosophy behind this decision to have different behaviors for the same method name?</p>
</blockquote>
</aside>
<p>My guess is the design was chosen for the sake of simplicity (iterate recursively through all registered modules, parameters, as well as buffers, and move them to the device) and to avoid high memory foot prints. I.e. a naive alternative approach would be to clone the model (2x memory usage) before moving the data to the device.</p>

dkleine: <p>Got it — thank you so much for your thorough and helpful response!</p>

ado_sar: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>If I understand correctly, the <code>tensor</code>’s of the module are modified in place (they have same id before and after the <code>.to</code> op), i.e. the underlying data and pointer are set to the <code>device</code>, and that’s it?</p>