Drawing gpu tensors to screen with pycuda/glumpy - updating depracted code

clbr: <p>Hey there <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:">!<br>
I’m currently experimenting around with using cuda and opengl (pycuda/glumpy) to draw tensors to the screen. I’ve found a gist that seems to have implemented exactly what i want to do. The problem is that the project is quite old and it uses a few deprecated apis.</p>
<p>The gist:</p><aside class="onebox githubgist" data-onebox-src="https://gist.github.com/victor-shepardson/5b3d3087dc2b4817b9bffdb8e87a57c4">
  <header class="source">

      <a href="https://gist.github.com/victor-shepardson/5b3d3087dc2b4817b9bffdb8e87a57c4" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://gist.github.com/victor-shepardson/5b3d3087dc2b4817b9bffdb8e87a57c4" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/victor-shepardson/5b3d3087dc2b4817b9bffdb8e87a57c4</a></h4>

  <h5>pytorch-glumpy.py</h5>
  <pre><code class="Python">from contextlib import contextmanager
import numpy as np
import torch
from torch import Tensor, ByteTensor
import torch.nn.functional as F
from torch.autograd import Variable
import pycuda.driver
from pycuda.gl import graphics_map_flags
from glumpy import app, gloo, gl
</code></pre>
    This file has been truncated. <a href="https://gist.github.com/victor-shepardson/5b3d3087dc2b4817b9bffdb8e87a57c4" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Being new to pytorch and machine learning in general, i was wondering if any of you guy’s have a few hints for me on how to get this working.</p>
<p>The packages i have installed:<br>
python - 3.9.7<br>
pytorch - 1.9.0+cu111<br>
pycuda - 2021.1+cuda114<br>
glumpy - 1.2.0</p>
<p>Since this is only doing inference there are quite a few <code>Variable(state, volatile=True)</code>. <code>Variable</code> seems to be deprecated, which is easily solved by doing the processing <code>with torch.no_grad():</code>.</p>
<p>What i’m struggling with is converting the tensor to the proper format of the texture. This is from line 99 onwards. Its mainly line 99 that currently throws <code>RuntimeError: t() expects a tensor with &lt;= 2 dimensions, but self is 3D</code>.</p>
<p>I’d appreciate any help.</p>
<p>Cheers</p>

ptrblck: <p>I don’t know what exactly changed in the code execution, but based on the <code>RuntimeError</code> it seems your tensors have now an additional dimension. <code>tensor.t()</code> would transpose a 2D tensor and you should be able to use <code>permute</code> instead on 3D tensors.<br>
However, it might also be a good idea to try to check why the number of dimension changed at all.</p>

clbr: <p>Appreciate the answer. I’m trying to understand how the tensors are being transformed. And how it relates to the outcome of what is being rendered in the end. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=10" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"><br>
Basically we have a texture of the shape <code>[512, 512, 4]</code>, meaning <code>[width, height, channels]</code>. I assume the channels are RGBA.<br>
And we want to go from a tensor with the shape <code>[1, 3, 512, 512]</code> and <code>[batch_size, channels, height, width]</code> to the format of the texture so it can be rendered. There is no alpha channel in this yet.</p>
<p>So please correct me if i’m wrong with this. I’d have to perform the following steps:</p>
<ol>
<li>
<code>tensor.squeeze()</code> the tensor to get rid of the batch_size, now we have <code>[3, 512, 512]</code> with <code>[channels, height, width]</code>
</li>
<li>
<code>tensor.transpose(0, 2)</code> to get <code>[512, 512, 3]</code> <code>[width, height, channels]</code>. My question here: what would’ve been the purpose of calling <code>tensor.t().data</code> on the output again? especially on a 3-dimensional tensor. I couldn’t find anything in the docs or changelogs of pytorch that any of the api’s used before this would’ve transformed the tensor into 2-D.</li>
<li>Add the alpha channel with <code>torch.cat((tensor, tensor[:, :, 0]), 2)</code> with <code>tensor</code> being in the same shape as the output of 2nd step. I still get the error: <code>Tensors must have same number of dimensions: got 2 and 3</code>. The input is obviously 3 dimensional, but <code>tensor[:, :, 0]</code> returns a tensor of the shape <code>[512, 512]</code>. I feel like this is the issue currently. <strong>Is there any ‘easy’ way of adding an extra column to all matrices that can represent the alpha channel?</strong> I want to go from shape <code>[512, 512, 3]</code> to <code>[512, 512, 4]</code>.</li>
<li>Set all values of the alpha channel to 1. <code>tensor[:, :, 3] = 1 </code>
</li>
</ol>
<p>Cheers</p>
<p><strong>Edit:</strong> Found the answer, can pretty much just do:</p>
<pre><code class="lang-auto">    a_tensor = torch.cuda.FloatTensor(h, w, 1)
    tensor = torch.cat((tensor, a_tensor), 2)
</code></pre>
<p>Is there a way to do this without having to create a tensor beforehand?</p>

ptrblck: <ol>
<li>Yes, but I would recommend to specify the dimension via <code>tensor.squeeze(0)</code> to make sure no other dimensions are removed in case they have a size of 1.</li>
<li>I don’t know which PyTorch version was originally used, but I thought <code>t()</code> did not work on &gt;2D data. In any case, use <code>tensor = tensor.permute(1, 2, 0)</code> to create the <code>[H, W, C]</code> tensor and don’t use the deprecated <code>.data</code> attribute.</li>
<li>You are indexing the tensor, which will remove this dimension and you could slice it instead:</li>
</ol>
<pre><code class="lang-python">tensor = torch.randn(4, 4, 3)
out = tensor[:, :, 0]
print(out.shape)
&gt; torch.Size([4, 4])

out = tensor[:, :, 0:1]
print(out.shape)
&gt; torch.Size([4, 4, 1])
</code></pre>
<p>However, I don’t think the alpha channel should just be the red channel from the same image, so you might want to use <code>torch.cat((torch.ones(H, W, 1), tensor), 2)</code> instead.</p>

clbr: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="132665" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>However, I don’t think the alpha channel should just be the red channel from the same image, so you might want to use <code>torch.cat((torch.ones(H, W, 1), tensor), 2)</code> instead.</p>
</blockquote>
</aside>
<p>Could you explain what you mean with <code> I don't think the alpha channel should just be the red channel from the same image</code>?</p>
<p>Using <code>torch.cat((torch.ones(h, w, 1), tensor), 2)</code> or <code>torch.cat((torch.cuda.FloatTensor(h, w, 1).fill_(1), tensor), 2)</code> to directly initialize the tensor on the gpu. I get something that seems like its completely tinted red.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/c/dced80e13e368f70c620beb186c0e0d4eae1ec89.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/dced80e13e368f70c620beb186c0e0d4eae1ec89" title="red_ting.PNG"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/c/dced80e13e368f70c620beb186c0e0d4eae1ec89.jpeg" alt="red_ting.PNG" data-base62-sha1="vwpTjRSUGdeGOcshJIHcGhVuZYd" width="472" height="500" data-dominant-color="9A545C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">red_ting.PNG</span><span class="informations">514×544 116 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>swapping the tensors passed into <code>torch.cat</code> around like <code>torch.cat((tensor, torch.cuda.FloatTensor(h, w, 1).fill_(1)), 2)</code> gives me the desired output.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a44b904983cfd29602a931bfd458cf81f374a78e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a44b904983cfd29602a931bfd458cf81f374a78e" title="normal.PNG"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a44b904983cfd29602a931bfd458cf81f374a78e.jpeg" alt="normal.PNG" data-base62-sha1="nrqcmKtntmHEAWHCnj12FA1a9ZA" width="472" height="500" data-dominant-color="9C868B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">normal.PNG</span><span class="informations">514×544 122 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I appreciate your help a lot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20">!</p>

ptrblck: <p>Yes, you are right, the alpha channel would be the last one (RGBA).</p>
<aside class="quote no-group" data-username="clbr" data-post="5" data-topic="132665">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/dc4da7/48.png" class="avatar"> clbr:</div>
<blockquote>
<p>Could you explain what you mean with <code> I don't think the alpha channel should just be the red channel from the same image</code>?</p>
</blockquote>
</aside>
<p>In your previous example you are reusing the red channels as the alpha channel, or rather you were concatenating the red channel as the first channel and pushed the original RGB channels back so that you would have this layout <code>[RRGB]=[RGBA]</code>. The alpha channel is not a clone of the red channel, but contains own information.</p>

clbr: <p>Yea that makes sense, thanks!</p>