Accessing the Single-Segment Buffer Interface

seba-1511: <p>Hi everyone,</p>
<p>I was wondering if there was a way to access the single-segment buffer interface of CUDA Tensors. For example, when using pyCUDA, I used to do:</p>
<pre><code>def buff(ary):
    return ary.gpudata.as_buffer(ary.nbytes)
</code></pre>
<p>I’d like to use mpi4py with Pytorch, and while I can use the numpy interface for CPU tensors, this is not possible for GPU ones. This would be a nice trick while we wait for the distributed interface to be polished.</p>
<p>Thanks a lot for your help (and for developing Pytorch),<br>
Séb</p>

smth: <p>you cannot use the buffer interface for CUDA tensors, but you can get the GPU pointer as an int:</p>
<pre><code class="lang-auto">x = torch.randn(10).cuda()
print(x.data_ptr())
</code></pre>

seba-1511: <p>Thanks for the fast answer.</p>
<p>Does that mean that I’d have to write my own C buffer wrapper ? (Maybe <code>PyBuffer_FromObject</code>: <a href="https://docs.python.org/2/c-api/buffer.html#c.PyBuffer_FromObject" rel="nofollow noopener">https://docs.python.org/2/c-api/buffer.html#c.PyBuffer_FromObject</a>)</p>
<p>I saw that in distributed you use <code>THDPTensorDesc</code> but I guess I can’t access it from Python for now. Do you have a suggestion as to what might be the best alternative ?</p>

smth: <p>do you really need a PyBuffer? We attempted implementing the Buffer interface, but it is slightly different across many versions of python and impossible to implement without thousands of lines of code.</p>
<p>Cant you work with the data pointer and size of tensor ?</p>

seba-1511: <p>The advantage of the Buffer interface is that mpi4py can take advantage of it. (Essentially making the whole of MPI available) I don’t know of a way to use it directly with address and length. Maybe using custom DataTypes, but I’d need to investigate that.</p>
<p>Since I can afford a hacky solution (only need send/recv), I’ll try that while waiting for THDP. In any case, I’ll keep this thread up-to-date.</p>

stsievert: <p>Any updates on this? I’m interested in using mpi4py with Tensors. Does Tensor use the PyBuffer interface?</p>

seba-1511: <p>Not that I am aware of. I (quickly) tried to implement a buffer interface at that time, but was not successful.</p>
<p>Nowadays, I would strongly recommend <code>torch.distributed</code>. It’s great and has support for MPI if you really need it.</p>

stsievert: <p>Does PyTorch work with MPI like other programs would? i.e., would the below work</p>
<pre><code class="lang-auto">$ mpiexec -n 4 python torch_script.py
</code></pre>
<p>when <code>torch_script.py</code> contains</p>
<pre><code class="lang-python">torch.distributed.init_process_group(backend='mpi', world_size=4)
</code></pre>

smth: <p>yes. that’s how we run our tests: <a href="https://github.com/pytorch/pytorch/blob/master/test/run_test.sh#L98-L108">https://github.com/pytorch/pytorch/blob/master/test/run_test.sh#L98-L108</a></p>

Stone: <p>To use MPI you have to build from source, I think it’s kind of annoying.</p>

seliad: <p>So, is there a workaround?</p>
<p>torch.distributed MPI operations are very limited (+requires building from source), while mpi4py supports more operations and seems like excellent library.</p>