Create/edit PyTorch tensor using OpenGL?

vargonis: <p>In order to create polygonal masks I’m currently using Pillow’s ImageDraw to draw them. Then, I can get the corresponding numpy arrays and upload to GPU. But I’m thinking about creating them directly on the GPU using OpenGL, via, say, pyglet or glumpy. I found somewhere else how to pass PyTorch tensors to CuPy using <code>data_ptr()</code> and the current CUDA stream, and I wonder whether something along those lines can be used to “draw” to a PyTorch tensor using OpenGL. Does anyone know how to do that?</p>

colesbury: <p>You can, but it’s going to require a bunch of work including C/C++ code.</p>
<ol>
<li>
<p>You need render to a texture using pyglet or glumpy. Get the texture ID (an int)</p>
</li>
<li>
<p>Use the CUDA OpenGL API to register the texture: <a href="https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__OPENGL.html" rel="nofollow noopener">https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__OPENGL.html</a></p>
</li>
<li>
<p>Use the CUDA Graphics API to bind the resource to a pointer. <a href="https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__INTEROP.html#group__CUDART__INTEROP" rel="nofollow noopener">https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__INTEROP.html#group__CUDART__INTEROP</a></p>
</li>
<li>
<p>Either cudaMemcpy that pointer into a Tensor’s data_ptr() or use torch::from_blob to wrap it as a Tensor.</p>
</li>
</ol>
<p>You’ll also need to call the appropriate unregister functions in destruction and make sure that the texture is in a matching format as the Tensor (i.e. float32 vs. int8)</p>

vargonis: <p>Could one avoid C/C++ coding for this with PyCUDA?</p>

colesbury: <p>I don’t think PyCUDA exposes the necessary functions. You might be able to use the Python <code>ctypes</code> library to call the functions, but it’s probably no easier than writing the actual C/C++ code.</p>

darknoon: <p>You might be interested in this gist, which uses a CUDA memcpy via PyCUDA to go the other direction. Presumably you could use the same technique to go from a texture to PyTorch.</p>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/victor-shepardson/5b3d3087dc2b4817b9bffdb8e87a57c4" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/victor-shepardson/5b3d3087dc2b4817b9bffdb8e87a57c4" target="_blank" rel="nofollow noopener">https://gist.github.com/victor-shepardson/5b3d3087dc2b4817b9bffdb8e87a57c4</a></h4>
<h5>pytorch-glumpy.py</h5>
<pre><code class="Python">from contextlib import contextmanager
import numpy as np
import torch
from torch import Tensor, ByteTensor
import torch.nn.functional as F
from torch.autograd import Variable
import pycuda.driver
from pycuda.gl import graphics_map_flags
from glumpy import app, gloo, gl
</code></pre>
This file has been truncated. <a href="https://gist.github.com/victor-shepardson/5b3d3087dc2b4817b9bffdb8e87a57c4" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


BarakChamo: <p>I’ve been following this thread and similar ones asking about device-to-device memory copy and interoperability with OpenGL and Cuda. I’ve also seen this great example <a class="mention" href="/u/darknoon">@darknoon</a> shared and was able to get it to run.</p>
<p>What I’m struggling to understand is at which point, given an OpenGL Texture or an existing Cuda array, do you take into consideration the data format the model is expecting and how the tensor needs to be constructed.</p>
<p>For example, YoloV3, which I’m trying to get to work on a GPU pipeline, expects a MCHW data format, but I don’t exactly understand how to find out the data format of my current Cuda surface or OpenGL texture, and how these can be copied over to the correct data format, or if that even matters?</p>

nicofirst1: <p>Is there any update on this?<br>
I’m working with deep reinforcement learning on the <a href="https://github.com/Farama-Foundation/PettingZoo" rel="noopener nofollow ugc">petting zoo</a> environment, specifically the <a href="https://www.pettingzoo.ml/mpe" rel="noopener nofollow ugc">MPE particle system</a>. I notice how the <a href="https://github.com/Farama-Foundation/PettingZoo/blob/aec87907777e0e62652d499b3ac96f680acf35ad/pettingzoo/mpe/_mpe_utils/rendering.py#L98" rel="noopener nofollow ugc">rendering function</a> in the system uses pyglet and OpenGL. It uses the latter to create the buffer image and then copies it to a numpy array which is pretty slow. I was wondering if you could grab the array directly from the GPU with a torch tensor. Is it possible without accessing C++ code?</p>

JonathanC: <p>I came to this thread looking for the answer as well, and figured it out by looking over the cuda-gl interop spec again. A little late I know, but here is a possible solution anyway: The approach I came up with is to first create a buffer object (VBO in my case) on the GL side, registering that with cuda. Then you map the registered buffer to get its device pointer, after which you can copy from your PyTorch <code>tensor.data_ptr()</code> to that device pointer with a cuda memcpy either dtod or htod depending on if the tensor is on gpu or cpu.</p>
<p>I implemented this in Python with pycuda which has a <code>pycuda.gl.RegisteredBuffer</code> class to register the GL buffer and has <code>.map()</code> and <code>.device_ptr_and_size()</code> to get the pointer. Finally, you can use <code>pycuda.driver.memcpy_h/dtod()</code> to copy the tensor over. If your tensor is on gpu this copy should be far faster than pulling the tensor into a numpy array and copying it to the buffer in GL manually, or similar.</p>

Cyanocitta: <p><a href="https://github.com/Ali-Asgari/CUDA_Tensor_Visualization_OpenGL" rel="noopener nofollow ugc">Ali-Asgari/CUDA_Tensor_Visualization_OpenGL: Real-time visualization of PyTorch tensors on CUDA with OpenGL (No transferring data between GPU and CPU)</a><br>
I think this guy solves this</p>