Move tensor to the same gpu of another tensor

username: Behrooz_Sepehry
cooked: <p>Assume I have a multi-GPU system. Let tensor “a” be on one of the GPUs, and tensor “b” be on CPU. How can I move “b” to the same GPU that “a” resides?</p>
<p>Unfortunately, b.type_as(a) always moves b to GPU 0.</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p><a href="http://pytorch.org/docs/0.3.1/tensors.html#torch.Tensor.new" rel="nofollow noopener">Tensor.new</a> will do exactly that.</p>

username: Behrooz_Sepehry
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>. The problem I have with the “Tensor.new” function is that</p>
<ol>
<li>
<p>If “a” is on GPU and “b” on CPU, then “a.new(b)” does not work (error: …constructor received an invalid combination of arguments…). “a.new(b.numpy())” works though, but I am afraid that it is inefficient.</p>
</li>
<li>
<p>If “a” and “b” are already on the same device, then “a.new(b)” will unnecessarily create a new copy of “b”</p>
</li>
</ol>
<p>I am looking for a function like “b.type_as(a)”, but could automatically move the data to the same device as “a”.</p>

username: ptrblck
cooked: <p>As far as I understand, you would like to move <code>b</code> to the same device as <code>a</code>.<br>
This should work:</p>
<pre><code class="lang-auto">a = torch.randn(10, 10).cuda()
print(a)
b = a.new(a)
print(b)
c = a.new(10, 10)
print(c)
</code></pre>

username: Behrooz_Sepehry
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> The problem is that “b” does not necessarily have the same shape and type of “a”. For example “a” could be 10⨉10 float tensor, while “b” 13⨉19⨉23 int Tensor.</p>

username: ptrblck
cooked: <p>You can pass the standard arguments to <code>new</code> as to a new <code>Tensor</code>:</p>
<pre><code class="lang-auto">a = torch.randn(10, 10).cuda()
print(a)
b = a.new(13, 19, 23).long()
print(b)
</code></pre>
<p>Would this work?</p>
<p>EDIT: You could of course pass a numpy array or something else to the constructor.<br>
Could you post your use case? I have the feeling, I’m not really understanding your problem and thus posting useless approaches. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"></p>

username: Behrooz_Sepehry
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>. I have a huge project, and in most places, in order to move the data to the proper device I used “type_as”. Then I wanted to run several instances of that program on different GPUs of the machine at the same time. The problem was that type_as always uses GPU 0. Right now I am using the approach explained here <a href="https://discuss.pytorch.org/t/select-gpu-device-through-env-vars/318">Select GPU device through env vars</a>, and it solves my problem.</p>

username: jxmorris12
cooked: <p>This may be new functionality from the Tensor API, but to move tensor <code>a</code> to the device of tensor <code>b</code> I use:</p>
<pre><code class="lang-auto">a = a.to(b.device)
</code></pre>

username: banikr
cooked: <p>The following also works irrespective of device:</p>
<pre><code class="lang-auto">a = a.to(b.get_device())
</code></pre>

username: TriKri
cooked: <p>I think you need to do <code>b = b.to(a.device)</code> in order to move <code>b</code> to the same GPU that <code>a</code> resides. Otherwise you will do the reverse.</p>