Copy raw float buffer to Tensor, efficiently, without numpy

jcowles: <p>Hey all,</p>
<p>I’m finding that our native file buffer reader can load data about 300x faster if we can skip a numpy ndarray copy. Unfortunately, all attempts to go directly to PyTorch seem to be orders of magnitude slower than going through numpy.</p>
<p>Is there any way I can memcpy (from Python) my data directly into a Tensor efficiently?</p>
<p>Stats:</p>
<pre><code class="lang-auto">Tensor:  90 seconds
torch.tensor(buffer)

Tensor Set_ + FloatStorage: 48 seconds
tensor.set_(FloatStorage(buffer))

TorchVision loader: 7 seconds

Numpy conversion: 0.7 seconds
torch.from_numpy(numpy.asarray(buffer)) # uses numpy protocol

Native loader: 0.002 seconds
</code></pre>
<p>While the Numpy version is quite a nice improvement, you can see the motivation to get our raw data into tensors directly and also skip Numpy.</p>
<p>I’m looking for something like this C++ code, but in Python:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b39bb0a6bd8bb3426e4c2982cd1898a0bfce5237.png" data-download-href="https://discuss.pytorch.org/uploads/default/b39bb0a6bd8bb3426e4c2982cd1898a0bfce5237" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b39bb0a6bd8bb3426e4c2982cd1898a0bfce5237.png" alt="image" data-base62-sha1="pCT1XrdjT5MbisAi6JPh9Dv7SrZ" width="690" height="342" data-dominant-color="F2F4F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">772×383 20.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://blog.christianperone.com/2018/03/pytorch-internal-architecture-tour/">
  <header class="source">

      <a href="https://blog.christianperone.com/2018/03/pytorch-internal-architecture-tour/" target="_blank" rel="noopener nofollow ugc">blog.christianperone.com</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://blog.christianperone.com/2018/03/pytorch-internal-architecture-tour/" target="_blank" rel="noopener nofollow ugc">PyTorch – Internal Architecture Tour | Terra Incognita</a></h3>

  <p>Update 28 Feb 2019: I added a new blog post with a slide deck containing the presentation I did for PyData Montreal. Introduction Short intro to Python extension objects in C/C++ Zero-copy PyTorch Tensor to Numpy and vice-versa Tensor Storage Shared...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>A similar request here:</p><aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/34646">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/34646" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/34646" target="_blank" rel="noopener nofollow ugc">Support creating a CPU tensor from ctypes pointer in Python / from_blob(ptr, shape, strides, dtype)</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-03-12" data-time="12:40:46" data-timezone="UTC">12:40PM - 12 Mar 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/vadimkantorov" target="_blank" rel="noopener nofollow ugc">
          <img alt="vadimkantorov" src="https://avatars.githubusercontent.com/u/1041752?v=4" class="onebox-avatar-inline" width="20" height="20">
          vadimkantorov
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: docs
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: memory usage
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          enhancement
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: numpy
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">NumPy has utilities that allow to create an array from a ctypes pointer in pure <span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">Python (without C++ extensions):
- https://docs.scipy.org/doc/numpy/reference/routines.ctypeslib.html#module-numpy.ctypeslib:
  - https://numpy.org/doc/stable/reference/routines.ctypeslib.html#numpy.ctypeslib.as_array
- https://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.ctypes.html

In C++ land, it seems that [`torch::from_blob`](https://pytorch.org/cppdocs/api/function_namespacetorch_1ad7fb2a7759ef8c9443b489ddde494787.html#function-documentation) should do the trick, but it has no binding to Python.

I propose to have utilities that would allow to do without passing by NumPy first or creating a C++ extension for a single-function `torch::from_blob` call, i.e. Python binding for `torch::from_blob` (that exist in java bindings btw), potentially with a custom deleter.

Context: I'd like to make a very thin ffmpeg audio-reading wrapper: the C code would do that allocation, return it to calling code, and the calling code would be responsible to free that memory. Pseudocode for NumPy (I still need to fix the C code) is here: https://github.com/vadimkantorov/audioprimer/blob/master/decode_audio.py . Ideally I'll have a single C file that's independent of NumPy/Torch and just slightly different versions of interfacing with it. I can think of some alternatives for this particular case, but Python way of creating a tensor from a raw ctypes pointer still may be useful for existing ctypes codebases.</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Following that last post, this code is the fastest pure-Torch implementation I can muster, however I’m not sure if it’s leaking memory and it’s still slower than numpy:</p>
<pre><code class="lang-auto">from_buffer: 1.1 seconds

t = torch.Tensor()
s = torch.Storage.from_buffer(buffer, byte_order="native")
t.set_(s)
</code></pre>
<p>Thanks,<br>
Jeremy</p>