Bitwise Operation on Float Tensor

Mafuyu_Aizawa: <p>Hello I’d like to ask how could I do bitwise operations on float tensors? Currently I have to use struct to convert them and then do the operation on CPU and move back, which is very inefficient… Do we have pytorch solutions for that?</p>
<p>Found previous questions like 6 years ago but no solutions there: <a href="https://discuss.pytorch.org/t/bitwise-operations-on-cuda-float-tensor/8173" class="inline-onebox">Bitwise Operations on Cuda Float Tensor</a></p>
<p>On the other hand, is it possible to interpret a tensor as another type (not type cast)? For example, interpret a DoubleTensor as a LongTensor? If I can do this, I can then use bitwise operations just like those for integer types…</p>

ptrblck: <p><code>view</code> should work as seen here:</p>
<pre><code class="lang-python">a = torch.tensor(1., device="cuda")
b = a.view(torch.int)
print(b)
# tensor(1065353216, device='cuda:0', dtype=torch.int32)

# both represent this bit pattern
# 00111111 10000000 00000000 00000000 
</code></pre>
<p>and allows you to interpret the data in another format.</p>

meghabyte: <p>Why doesn’t this work with doubles? When I try torch.float64, I don’t get matching bit patterns.</p>

ptrblck: <p>Could you post an example showing the issue in the same way I’ve shown it works?</p>

meghabyte: <aside class="quote no-group" data-username="Mafuyu_Aizawa" data-post="1" data-topic="170863">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mafuyu_aizawa/48/14255_2.png" class="avatar"> Mafuyu_Aizawa:</div>
<blockquote>
<p>Bitwise Operations on Cuda Float Tensor</p>
</blockquote>
</aside>
<p>Never mind, I think I resolved the issue. I was manually comparing with the results from these two IEEE754 converters from double to binary, but there are slight differences (e.g. the closest float to 3.1415):</p>
<ul>
<li><a href="https://www.binaryconvert.com/result_double.html" class="inline-onebox" rel="noopener nofollow ugc">Double (IEEE754 Double precision 64-bit)</a></li>
<li><a href="https://www.h-schmidt.net/FloatConverter/IEEE754.html" class="inline-onebox" rel="noopener nofollow ugc">IEEE-754 Floating Point Converter</a></li>
</ul>
<p>The former seems to match what .view(torch.long) gives.</p>

Geremia: <p>Is there a way to preserve the grad_fn with view?</p>

ptrblck: <p>Sorry, I don’t understand the question so could you explain how the bit representation, which can be changed via the <code>view</code> operation, being discussed in this topic is related to a <code>grad_fn</code>?</p>

Geremia: <p>For example, doing <code>view(torch.int)</code> on a <code>tensor(6234., grad_fn=&lt;DotBackward0&gt;)</code> makes it lose the <code>grad_fn</code>: <code>tensor(1170395136, dtype=torch.int32)</code></p>
<p>Never mind, it isn’t possible, as “only Tensors of floating point dtype can require gradients”.</p>

ptrblck: <p>Thanks for the explanation. Interpreting the tensor in another <code>dtype</code> is not differentiable (e.g. integer <code>dtypes</code> are not differentiable by design) so you would need to implement your <code>backward</code> function manually based on your definition of the backward operation.</p>

Geremia: <aside class="quote no-group" data-username="ptrblck" data-post="9" data-topic="170863">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>you would need to implement your <code>backward</code> function manually based on your definition of the backward operation.</p>
</blockquote>
</aside>
<p>Like <a href="https://discuss.pytorch.org/t/bitwise-operations-on-cuda-float-tensor/8173/5">this</a>?</p>

ptrblck: <p>Almost, as I don’t think directly assigning tensors to <code>ctx</code> and unpacking them is a valid approach since it could cause memory leaks, if I’m not mistaken.<br>
Use <code>ctx.save_for_backward</code> and <code>ctx.saved_tensors</code> instead as shown in the <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">tutorial</a>.</p>