Can torch.Tensor have different precision element-wisely?

FruitVinegar: <p>Hello. pytorch users.<br>
Like title, can torch.Tensor have different precision element-wisely?<br>
for example:</p>
<pre><code class="lang-auto">my_tensor = torch.Tensor([1, 1.0, 1.000])
# respectively types are: torch.int8, torch.half, torch.float... etc.
</code></pre>
<p>If it is impossible(maybe I think pytorch doesn’t support this semantic), is there another way to implement like this? I’d like to reduce memory usage as much as I can…</p>
<p>Any answers will be helpful for me. Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>I don’t think this is possible using a tensor, since e.g. the strides would be hard (impossible) to handle.<br>
Python containers (e.g. <code>list</code>) are quite flexible regarding different types of elements, but you won’t be able to apply e.g. a matrix multiplication on this “array”.</p>
<p>If you are running out of memory, you could have a look at <a href="https://pytorch.org/docs/stable/checkpoint.html" rel="nofollow noopener"><code>torch.utils.checkpoint</code></a> to trade compute for memory or e.g. <a href="https://github.com/NVIDIA/apex" rel="nofollow noopener">apex/amp</a> for mixed precision training.</p>

FruitVinegar: <p>Thanks for the answer.<br>
I’m back here to share my progressed idea to this problem.</p>
<p>Back to the problem, ‘element-wise(pixel-wise) different precision in Tensor(Image)’.</p>
<p>For example, look at this picture:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0fbe5a44185e7a2e04c7c8fe8149082fe65c288e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/0fbe5a44185e7a2e04c7c8fe8149082fe65c288e" title="light-4297386_1920"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0fbe5a44185e7a2e04c7c8fe8149082fe65c288e_2_690x387.jpeg" alt="light-4297386_1920" data-base62-sha1="2fgZpXB04t5R1ccW79s4KZ0RO2i" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0fbe5a44185e7a2e04c7c8fe8149082fe65c288e_2_690x387.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0fbe5a44185e7a2e04c7c8fe8149082fe65c288e_2_1035x580.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0fbe5a44185e7a2e04c7c8fe8149082fe65c288e_2_1380x774.jpeg 2x" data-dominant-color="7B6A40"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">light-4297386_1920</span><span class="informations">1920×1079 315 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Although the background of image is not-focused well, human can recognize the object in this Image is lamp.<br>
On a similar principle, can object-detection networks recognize objects from Image that has different precision(resolution) pixel-wisely? Moreover, can we get memory reduction effect from this solution?</p>
<p>This is why I suggested this problem.</p>
<p>From now, this is my progressed idea.<br>
I made an helper function from this <a href="https://discuss.pytorch.org/t/how-to-know-the-memory-allocated-for-a-tensor-on-gpu/28537">forum</a></p>
<pre><code class="lang-auto"># returns actual allocated memory size of given tensor
def get_actual_memsize(tensor):
    return tensor.element_size() * tensor.nelement()
</code></pre>
<p>To test my idea, I made a sample tensor:</p>
<pre><code class="lang-auto">B, C, H, W = 1, 10, 224, 224

tensor = torch.randn(B, C, H, W)

print(tensor.type())
print(get_actual_memsize(tensor))
</code></pre>
<p>this returns:</p>
<blockquote>
<p>torch.FloatTensor<br>
2007040</p>
</blockquote>
<p>I decomposed tensor based on each element’s value, like this:</p>
<pre><code class="lang-auto">threshold = 0.2

important_region = torch.ge(tensor, threshold)
unimportant_region = ~important_region #just reverse

unimportant_values = torch.masked_select(tensor, unimportant_region).to(torch.float16) #Precision Reduction
important_values = torch.masked_select(tensor, important_region)
</code></pre>
<pre><code class="lang-auto">print(unimportant_values.type())
print(important_values.type())

print(get_actual_memsize(unimportant_values) + get_actual_memsize(important_values) + get_actual_memsize(unimportant_region))
</code></pre>
<p>and memory reduced: 2007040 → 1927504</p>
<blockquote>
<p>torch.HalfTensor<br>
torch.FloatTensor<br>
1927504</p>
</blockquote>
<p>Since the sample tensor is small(size is [1, 10, 244, 244]), the reducing effect is much more larger in bigger tensor, I think.</p>
<p>Now I’m restoring tensor with these decomposed components. It’s quite confusing. I’m looking for suitable pytorch API now(like masked_fill) but all I found is not suitable…</p>
<p>This is my current idea. and if you have a similar idea or knowledge about pytorch API, please answer how to restoring the tensor please.</p>

ptrblck: <aside class="quote no-group" data-username="FruitVinegar" data-post="3" data-topic="55195">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fruitvinegar/48/39246_2.png" class="avatar"> FruitVinegar:</div>
<blockquote>
<p>Now I’m restoring tensor with these decomposed components. It’s quite confusing. I’m looking for suitable pytorch API now(like masked_fill) but all I found is not suitable…</p>
<p>This is my current idea. and if you have a similar idea or knowledge about pytorch API, please answer how to restoring the tensor please.</p>
</blockquote>
</aside>
<p>That’s the pitfall I tried to mention in my last post.<br>
I’m not sure, if there might be a clean way to reconstruct e.g. an array (tensor) with mixed precision data types, as the strides would change based on the current element type.</p>
<p>Let’s think about a plain tensor:</p>
<pre><code class="lang-auto">x = torch.tensor([[0., 1., 2., 3.],
                  [4., 5., 6., 7.]])
print(x.size())
&gt; torch.Size([2, 4])
print(x.stride())
&gt; (4, 1)
print(x.nelement())
&gt; 8
print(x.element_size())
&gt; 4
</code></pre>
<p>The stride shows, how many <code>element_size()</code> bytes you would have to skip in the contiguous tensor to get to the next index in the corresponding dimension.<br>
E.g. to index the <code>5</code> in <code>x</code>, you would have to use <code>x[1, 1]</code>, which corresponds to <code>4+1</code> elements offset from the beginning of the data.<br>
Also, as you can see the <code>element_size()</code> is defined per tensor, which makes the indexing (and shape changes) via strides possible.<br>
If you are using different data types in the tensor, you would e.g. need to access the size of each elements to index any value in it.</p>
<p>That being said, this approach won’t work for this strided approach using contiguous memory chunks. There might be some other data layout, I’m not aware of, so please let me know, if you find something.</p>