Efficiently converting np.uint8 to torch.float32

xvdp: <p>I was running some data ffmpeg to torch (thru pipes) and noticed that I was doing something very naive. So I profiled, with single process, conversion between npuint to float32<br>
The difference in CPU can be almost one order of magnitude. GPU tightens everything.</p>
<p>If anyone is interested.<br>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/xvdp/149e8c7f532ffb58f29344e5d2a1bee0" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/xvdp/149e8c7f532ffb58f29344e5d2a1bee0" target="_blank" rel="nofollow noopener">https://gist.github.com/xvdp/149e8c7f532ffb58f29344e5d2a1bee0</a></h4>
<h5>npuint8_torchfloat32.py</h5>
<pre><code class="Python">""" I was writing a dataloader from a video stream. I ran some numbers.
# in a nutshell. 
-&gt; np.transpose() or torch.permute() is faster as uint8, no difference between torch and numpy
-&gt; np.uint8/number results in np.float64, never do it, if anything cast as np.float32
-&gt; convert to pytorch before converting uint8 to float32
-&gt; contiguous() is is faster in torch than numpy
-&gt; contiguous() is faster for torch.float32 than for torch.uint8
-&gt; convert to CUDA in the numpy to pytorch conversion, if you can.

When loading a dataset a quite typical operation is to load the data - which may come thru numpy -</code></pre>
This file has been truncated. <a href="https://gist.github.com/xvdp/149e8c7f532ffb58f29344e5d2a1bee0" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

ptrblck: <p>Thanks for sharing the code!</p>
<p>Be a bit careful about the CUDA numbers, since you didn’t synchronize the calls.<br>
CUDA calls will be executed asynchronously, which means the CPU can continue executing the code while the CUDA kernels are busy until a synchronization point is reached.<br>
These points can be added manually using <code>torch.cuda.synchronize()</code> are are automatically reached, e.g. if a result from the CUDA operation is needed.<br>
In your profiling script you might in fact just time the kernel launch times, when the <code>.cuda()</code> call is the first op on the tensor.</p>
<p>To proper time CUDA calls, you could have to synchronize before starting and stopping the timer:</p>
<pre><code class="lang-python">torch.cuda.synchronize()
t0 = time.time()
...
torch.cuda.synchronize()
t1 = time.time()
</code></pre>

xvdp: <p>Thank you ptrblck! You are right I didnt sync the cuda.  Ill update the gist when I get to this. I also wonder whether this profiling is at all significant when using multiprocess multi gpu. I need to update my hw setup anyway which as you see is from the paleozoic.</p>
<p>I wrote this gist mainly because I had ndarrayuint8/255 -&gt; torch (float32 cuda) peppered all over my code which is a waste of resources and time. I figured someone else will be doing the same stupid mistake.</p>

ptrblck: <p>Sure and thanks again for posting!<br>
I just skimmed through your code and will dig in it a bot later, as I’m interested in the results <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

xvdp: <p>great - if you look at this, there are a few things I didn’t post; float64 or float16 torch data, but it varies wildly.  I dont have a good setup to deal with double or half floats so the comparison isnt valid.</p>
<p>I found some funny numbers to do with contiguity which I also didn’t post, because first it worked one way then another. But it went something like this,</p>
<pre><code class="lang-python">tensor.permute(2,1,0).contiguous()
# vs
w,h,c = tensor.shape
out =  torch.zeros([1,c,h,w]])
out[0] = tensor.permute(2,1,0)
# both return a contiguous tensor
</code></pre>
<p>in some instances i got the latter to be 2x the speed of the ‘proper way’<br>
although, maybe this is meaningless. Because the test wasn’t consistent.</p>
<p>At any rate if one is serious about the speed one should probably be working in cpp. Which I thought, maybe the way to get rid of all this uncertainty is load from buffer to a torch tensor of shape dtype device that is contiguous, directly in cpp… But thats another day.</p>

ptrblck: <p>I don’t think these operation see any significant speedup in C++, if the tensors are “large enough” (i.e. not tiny).<br>
Usually you would just see the Python overhead, which is negligible if you have some workload on the operations.</p>
<p>Both methods should yield a similar timing and I would assume the second approach to be slower.<br>
At least, that’s the result from my profiling:</p>
<pre><code class="lang-python">x = torch.randn(64, 128, 64, 64, device='cuda')
nb_iters = 1000

# warmup
for _ in range(100):
    y = x.permute(0, 3, 2, 1).contiguous()

torch.cuda.synchronize()
t0 = time.time()

for _ in range(nb_iters):
    y = x.permute(0, 3, 2, 1).contiguous()

torch.cuda.synchronize()
t1 = time.time()

print((t1 -t0)/nb_iters)


# warmup
for _ in range(100):
    out =  torch.zeros(64, 64, 64, 128, device='cuda')
    out[:] = x.permute(0, 3, 2, 1)

torch.cuda.synchronize()
t0 = time.time()

for _ in range(nb_iters):
    out =  torch.zeros(64, 64, 64, 128, device='cuda')
    out[:] = x.permute(0, 3, 2, 1)

torch.cuda.synchronize()
t1 = time.time()

print((t1 -t0)/nb_iters)
</code></pre>
<p>Permute + contiguous: 0.00072478 s<br>
Permute + copy: 0.00093216 s</p>