Loading an unsigned 8bit integer

Kore_ana: <p>Hello! Is it possible to load an 8 bit unsigned integer as a 8 bit float between 0~1(?)(if it exists).</p>
<p>I have a data that is inherently an 8bit unsigned integer (0~255), but I want to normalize it to 0~1 before performing the forward pass. I guess there would be two ways to do this :</p>
<ol>
<li>since torch tensor seems to support 8 bit unsigned integers, load the 8 bit unsigned integer to the gpu then normalize it inside of the GPU</li>
<li>normalize the 8bit integer while inside the CPU tensor (convert it to float) then load that to the GPU</li>
</ol>
<p>I have a question about the second option. It seems that there isn’t anything like “8 bit integer” or equivalent in float. Is there? Is there a way to convert the 8 bit integer into an 8 bit float tensor (so that the precision is preserved?)</p>
<p>I am asking  because I would prefer to do 2. instead of 1. because that would make the code cleaner!</p>
<p>(sorry if my questions are a bit basic <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> )</p>

ptrblck: <p>The second approach is the common one and an <code>uint8</code> image tensor will be normalized to a <code>float32</code> tensor in the range <code>[0, 1]</code> using <code>torchvision.transforms.ToTensor()</code> as seen in this example:</p>
<pre data-code-wrap="python"><code class="lang-python"># load uint8 image
img = PIL.Image.open(PATH)
# or generate random image
img = transforms.ToPILImage()(torch.randn(3, 224, 224))
print(np.array(img).min(), np.array(img).max())
# 0 255

out = transforms.ToTensor()(img)
print(out.min(), out.max())
# tensor(0.) tensor(1.)
</code></pre>
<aside class="quote no-group" data-username="Kore_ana" data-post="1" data-topic="176627">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kore_ana/48/45740_2.png" class="avatar"> Kore_ana:</div>
<blockquote>
<p>Is there a way to convert the 8 bit integer into an 8 bit float tensor (so that the precision is preserved?)</p>
</blockquote>
</aside>
<p>8bit integers, i.e. images using the <code>uint8</code> data type, can be mapped to <code>float32</code> without a loss in precision, since <code>float32</code> can represent all integers up to <code>2**24</code> where a rounding to multiple of 2s starts:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.tensor(2.**24)
# tensor(16777216.)
torch.tensor(2.**24+1)
# tensor(16777216.)
torch.tensor(2.**24+2)
# tensor(16777218.)
</code></pre>
<p>in case you want to keep the original integer values. If not, you can also verify that a “reverse” mapping is possible:</p>
<pre><code class="lang-auto">y = out * 255
(y == torch.from_numpy(np.array(img)).permute(2, 0, 1)).all()
# tensor(true)
</code></pre>

Kore_ana: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for your quick reply!</p>
<p>I guess I will try the second option! However, I wanted to ask : wouldn’t changing an 8 bit integer to 32 bit float mean that during data loading 4 times the data has to be loaded to the GPU?</p>

ptrblck: <p>Yes, but also all model parameters are stored in <code>float32</code> and the gradient calculation needs floating point values. You could use mixed-precision training as described <a href="https://pytorch.org/docs/stable/notes/amp_examples.html">here</a> which would allow you to use <code>float16</code> or <code>bfloat16</code> data types, where it’s considered to be safe.<br>
Lower numerical formats are currently being developed and e.g. <a href="https://github.com/NVIDIA/TransformerEngine"><code>TransformerEngine</code></a> allows you to use some Transformer-specific modules in <code>FP8</code> format on the latest NVIDIA Hopper GPU architecture.</p>