Reinterpret PyTorch array as a different dtype

wjakob: <p>Dear all,</p>
<p>I’m looking for a way of reinterpreting a PyTorch tensor as a different <code>dtype</code> of matched size. This kind of operation is provided by NumPy, but as far as I can tell PyTorch only provides casts. (naturally, reinterpreting types is an operation that would only make sense for tensors without gradients).</p>
<p>Here is an example of what I mean, using NumPy:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = np.linspace(0, 1, 5, dtype=np.float32)
&gt;&gt;&gt; a
array([ 0.  ,  0.25,  0.5 ,  0.75,  1.  ], dtype=float32)

&gt;&gt;&gt; a.view(np.int32)
array([         0, 1048576000, 1056964608, 1061158912, 1065353216], dtype=int32)
</code></pre>
<p>My current workaround involves a PyTorch -&gt; NumPy -&gt; PyTorch roundtrip, which is not very nice/efficient.</p>
<pre><code class="lang-auto">
&gt;&gt;&gt; a = torch.linspace(0, 1, 5)
&gt;&gt;&gt; torch.Tensor(a.numpy().view(dtype=np.int32))
tensor([         0., 1048576000., 1056964608., 1061158912., 1065353216.])
</code></pre>
<p>I’d be very grateful for any suggestions on how PyTorch could be convinced to change the <code>dtype</code> for an already-created memory region.</p>
<p>Thanks,<br>
Wenzel</p>

SimonW: <p>I’m curious. How is <code>np.view(dtype)</code> generally used?</p>

ptrblck: <p>Probably not a general use case, but I’ve seen it being used in <a href="https://github.com/numpy/numpy/blob/5adb81051086a45fe3b59ba506b567340ec9bd5e/numpy/lib/arraysetops.py#L250">np.unique</a> to consolidate the reshaped array.</p>

wjakob: <p>These kinds of type reinterpretations are an important building block for implementing special functions. It’s quite common to switch from float to integer, do a few manipulations there (e.g. of the mantissa), and then switch back to floats in the end.  In the NumPy world, <code>np.view</code> is the operation that enables one to do that. AFAIK it’s not possible with PyTorch. There is no analog of <code>np.view</code> as far as I can see, and directly assigning the <code>dtype</code> yields an error message:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; t.dtype = torch.int32
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
AttributeError: attribute 'dtype' of 'torch._C._TensorBase' objects is not writable
</code></pre>
<p>Or do you have any suggestions?</p>

SimonW: <p>If you are writing a C++ extension function, you can use <code>reinterpret_cast&lt;int64_t*&gt;(tensor.data&lt;float&gt;())</code>. But unfortunately I don’t think that this is well supported in Python. Please file a feature request on GitHub. Thanks!</p>

yurib: <p>Hello, all!</p>
<p>Wonder if feature request has been filed? Another possible usage (for the lack of better option?) - cpu HalfTensor doesn’t support loads of operations, even as basic as index() &amp; index_select(), which are useful for slicing batch from large (regressor) matrix (index_select is  also very efficient/fast)<br>
So one workaround is ‘pack’ 2 float16 columns to one float32 (or even 4 float16 to one float64), then select required rows (for next batch), then ‘unpack’ back to float16</p>
<p>numpy(), from_numpy() trick works but looks ugly</p>

calvinmccarter: <p>For those curious, this feature has been implemented and merged: <a href="https://github.com/pytorch/pytorch/pull/47951" class="inline-onebox" rel="noopener nofollow ugc">Add tensor.view(dtype) by zasdfgbnm · Pull Request #47951 · pytorch/pytorch · GitHub</a></p>
<p>So now you can use <code>tensor.view(dtype=...)</code>.</p>