akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Dose SyncBN with DDP support different data size in GPUs

wbhu: <p>Hi,</p>
<p>I am working with a scripts that each training instance has different mumber of points, so when I applied DDP with syncBN, how will the BN stats in different GPUs to sync? I have traced the source code to here “<a href="https://github.com/pytorch/pytorch/blob/a4a5b6fcaae26fe241d32a7c4b2091ee69b600bb/torch/nn/modules/_functions.py" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/a4a5b6fcaae26fe241d32a7c4b2091ee69b600bb/torch/nn/modules/_functions.py</a>”  L33-L43</p>
<pre><code class="lang-auto">        # calcualte global mean &amp; invstd
        mean, invstd = torch.batch_norm_gather_stats_with_counts(
            input,
            mean_all,
            invstd_all,
            running_mean,
            running_var,
            momentum,
            eps,
            count_all.view(-1).long().tolist()
        )
</code></pre>
<p>In my case the “mean_all” and “invstd_all” should be weighted average accroding to different “counts” in GPUs, is it the actual situation?</p>
<p>BTW, the syncBN in NVIDIA apex just simply average “mean_all” and “invstd_all” which not support for different counts in GPUs.</p>
<p>Thanks very much</p>

elmirador: <blockquote>
<p>In my case the “mean_all” and “invstd_all” should be weighted average according to different “counts” in GPUs, is it the actual situation?</p>
</blockquote>
<p>I think you’re right.</p>
<p><code>torch.batch_norm_gather_stats_with_counts</code> leads to <code>aten\src\ATen\native\cuda\Normalization.cuh</code>.</p>
<p>The function you’re finding is <code>batch_norm_reduce_statistics_kernel</code>.<br>
In the loop starts at L405, you can see that all statistics are calculated w.r.t. their own <code>count</code>.</p>
<pre><code> for (int j = 0; j &lt; world_size; j++) {
  scalar_t count = counts[j];
  accscalar_t m = vec_mean[j][i];
  accscalar_t v = accscalar_t(1.0) / (vec_invstd[j][i]);
  v = (v * v - epsilon) * count;
  accscalar_t factor = 1.0 / (n + count);
  var_n += v + (avg - m) * (avg - m) * n * count * factor;
  avg = n * factor * avg + count * factor * m;
  n += count;
}
</code></pre>
What is the optimal way to train model Right to left

Aiman_Mutasem-bellh: <p>Hello everyone <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I’m using standard TRANSFORMER for NMT and I’m going to train model Right to left and I have two ideas:</p>
<ol>
<li>
<p>reversing the input text from left to Right, before feeding the data to the encoder and decoder.</p>
</li>
<li>
<p>reversing the embedding vector in the encoder and decoder.</p>
</li>
</ol>
<p>My question is what the optimal way to train model Right to Left?</p>
<p>Note: I have two versions of my model  (RTL model) and (LTR model).</p>
Decode a PNG-encoded image

dachosen1: <p>Does Pytorch have a function to decode a PNG image to a uint8 or uint16 tensor? I’m trying to track confusion matrix in tensorboard, and I found a way in Tensorflow to that converts the matplotlib plot specified by ‘figure’ to a PNG image, wanted to see if there was an equivalent for Pytorch.</p>

ptrblck: <p>I’m not sure to understand the question completely, so please feel free to correct my understanding.</p>
<aside class="quote no-group" data-username="dachosen1" data-post="1" data-topic="77231">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/47e85d/48.png" class="avatar"> dachosen1:</div>
<blockquote>
<p>Does Pytorch have a function to decode a PNG image to a uint8 or uint16 tensor?</p>
</blockquote>
</aside>
<p>PyTorch, or rather <code>torchvision</code>, uses <code>PIL</code> as the default library to load and manipulate images.<br>
<code>PIL</code> is able to load <code>PNG</code> files and the numpy array can be transformed to a tensor afterwards.</p>
<aside class="quote no-group" data-username="dachosen1" data-post="1" data-topic="77231">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/47e85d/48.png" class="avatar"> dachosen1:</div>
<blockquote>
<p>I found a way in Tensorflow to that converts the matplotlib plot specified by ‘figure’ to a PNG image, wanted to see if there was an equivalent for Pytorch.</p>
</blockquote>
</aside>
<p>You can store <code>matplotlib</code> images directly as <code>PNG</code>s using <code>matplotlib.pyplot.savefig</code>.</p>

dachosen1: <p>gotcha. Thanks for the help</p>
GAN training - few examples

therohanpandey: <p>Does it make sense to train a WGAN/ or any GAN network for that matter on examples as few as 100? In order to augment the data set, catering to data imbalance. Does it make sense or will I just be creating the same examples again? (I ran a WGAN with GP for 5000 iterations) Will it overfit the data?</p>
CNN training multilabel clasiffication error

jlaj: <p>Hi,</p>
<p>I am getting an error while training this CNN, the inputs are images and the labes are one-hot encoded vectors, and I am getting the following error when training the CNN</p>
<p>Thanks for the help in advance<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/1/41647b9cd9240b58a7d35e24085268455ef42f12.png" data-download-href="https://discuss.pytorch.org/uploads/default/41647b9cd9240b58a7d35e24085268455ef42f12" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41647b9cd9240b58a7d35e24085268455ef42f12_2_690x373.png" alt="image" data-base62-sha1="9kujMGOYelTiUxZzla0xsWoszUm" width="690" height="373" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41647b9cd9240b58a7d35e24085268455ef42f12_2_690x373.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41647b9cd9240b58a7d35e24085268455ef42f12_2_1035x559.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41647b9cd9240b58a7d35e24085268455ef42f12_2_1380x746.png 2x" data-dominant-color="F3F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2777×1505 418 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> expects the target as a <code>LongTensor</code> and to have the shape <code>[batch_size]</code> containing the class indices in the range <code>[0, nb_classes]</code>.<br>
Since your target is one-hot encoded, you could use:</p>
<pre><code class="lang-python">target = torch.argmax(target, dim=1)`
</code></pre>
<p>to create the expected shape and type.</p>
<p>PS: It’s better to post code snippets by wrapping them into three backticks ``` instead of screenshots. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Minimum version required for CUDA

choidongyeon: <p>Is there an easy way in the code base to detect the minimum version of CUDA that is required for any specific version of PyTorch?</p>

ptrblck: <p>Would you like to check if in the code before building?<br>
Note that you won’t be able to build PyTorch with a CUDA version, which is too old (e.g. CUDA8 for the current master), and the binaries won’t thus be built for this version.</p>
Simple timing question when casting to float on GPU

Muppet: <p>Hi guys,</p>
<p>How efficient is the following?</p>
<pre><code class="lang-auto">rewards = torch.as_tensor(rewards).to(self.device).float()
</code></pre>
<p>vs</p>
<pre><code class="lang-auto">rewards = torch.as_tensor(rewards).float().to(self.device)
</code></pre>
<p>What is the best way to benchmark that given all the randomness with GPU?</p>
<p>Lastly, if a tensor x on the GPU is already a float and then applying another x = x.float() operation - is that efficient? In other words, does PyTorch know it’s already float and will not do the cast? Or does that result in a significant overhead that should be avoided?</p>
<p>Thanks!</p>

KFrank: <p>Hi Muppet!</p>
<aside class="quote no-group" data-username="Muppet" data-post="1" data-topic="77222" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/53a042/48.png" class="avatar"> Muppet:</div>
<blockquote>
<pre><code class="lang-auto">rewards = torch.as_tensor(rewards).to(self.device).float()
</code></pre>
<p>vs</p>
<pre><code class="lang-auto">rewards = torch.as_tensor(rewards).float().to(self.device)
</code></pre>
</blockquote>
</aside>
<p>Speaking as a non-expert, I would think that</p>
<pre><code class="lang-auto">rewards = torch.as_tensor (rewards, dtype = torch.float, device = torch.device('cuda'))
</code></pre>
<p>would be the way to go.  I believe that pytorch will “no-op” any<br>
unneeded casts.</p>
<p>Note, if <code>rewards</code> starts out as, for example, a numpy double array<br>
(and <code>self.device</code> is the gpu), I think converting to float first, and<br>
then moving it to the gpu would win, because bandwidth to the gpu<br>
is a bottleneck, and moving the float would only involve half as<br>
much data as moving the double.</p>
<aside class="quote no-group">
<blockquote>
<p>does PyTorch know it’s already float and will not do the cast? Or does that result in a significant overhead that should be avoided?</p>
</blockquote>
</aside>
<p>I’m pretty sure that pytorch knows enough to skip redundant casts.<br>
(May the experts correct me if I’m wrong.)</p>
<p>Best.</p>
<p>K. Frank</p>

Muppet: <p>Thanks for the reply. Yeah it would be really interesting if someone can confirm this for sure?</p>

ptrblck: <p>Yes, as <a class="mention" href="/u/kfrank">@KFrank</a> explained, no copy/transform will be triggered, if the data is already in the desired type.<br>
Also <code>as_tensor</code> will share the data of the underlying numpy array (if you leave it on the CPU):</p>
<pre><code class="lang-python">x = np.zeros((2, 2))
y = torch.as_tensor(x, dtype=torch.float64)
print(x, '\n', y)

y[0, 0] = 1.
print(x, '\n', y)

# Changing the data type will create a copy
x = np.zeros((2, 2))
y = torch.as_tensor(x, dtype=torch.float32)
print(x, '\n', y)

y[0, 0] = 1.
print(x, '\n', y)
</code></pre>
Replacing torch.cdist() function to eliminate GPU out-of-memory runtime error

promach: <p>How shall I modify this <a href="https://github.com/huawei-noah/AdderNet/issues/6#issuecomment-594212162" rel="nofollow noopener">new_cdist()</a> function to eliminate <a href="https://github.com/huawei-noah/AdderNet/issues/6#issuecomment-613362421" rel="nofollow noopener">GPU out-of-memory runtime error</a> ?</p>
<p>More specifically replacing <code>torch.cdist()</code> with something lighter in memory footprint.</p>
<p>Note: The following code snippet is related to a <a href="https://github.com/huawei-noah/AdderNet/issues/5" rel="nofollow noopener">new type of back-propagation equation and adaptive learning rate</a>.</p>
<p>See <a href="https://stackoverflow.com/questions/61154470/understanding-cdist-function" rel="nofollow noopener">here</a> for more information on how the existing code works</p>
<p>I saw there are two cdist() implementations online (<a href="https://github.com/pytorch/pytorch/pull/25799#issuecomment-529021810" rel="nofollow noopener">code 1</a> , <a href="https://github.com/pytorch/pytorch/issues/15253#issuecomment-491467128" rel="nofollow noopener">code 2</a>)</p>
<pre><code class="lang-auto">def new_cdist(p, eta):
    class cdist(torch.autograd.Function):
        @staticmethod
        def forward(ctx, W, X):
            ctx.save_for_backward(W, X)
            out = -torch.cdist(W, X, p)
            return out

        @staticmethod
        def backward(ctx, grad_output):
            W, X = ctx.saved_tensors
            grad_W = grad_X = None
            if ctx.needs_input_grad[0]:
                _temp1 = torch.unsqueeze(X, 2).expand(X.shape[0], X.shape[1], W.shape[0]).permute(1, 0, 2)
                _temp2 = torch.unsqueeze(W.transpose(0, 1), 1)
                _temp = torch.cdist(_temp1, _temp2, p).squeeze().transpose(0, 1)
                grad_W = torch.matmul(grad_output, _temp)
                # print('before norm: ', torch.norm(grad_W))
                grad_W = eta * np.sqrt(grad_W.numel()) / torch.norm(grad_W) * grad_W
                print('after norm: ', torch.norm(grad_W))
            if ctx.needs_input_grad[1]:
                _temp1 = torch.unsqueeze(W, 2).expand(W.shape[0], W.shape[1], X.shape[0]).permute(1, 0, 2)
                _temp2 = torch.unsqueeze(X.transpose(0, 1), 1)
                _temp = torch.cdist(_temp1, _temp2, p).squeeze().transpose(0, 1)
                _temp = torch.nn.functional.hardtanh(_temp, min_val=-1., max_val=1.)
                grad_X = torch.matmul(grad_output.transpose(0, 1), _temp)
            return grad_W, grad_X
    return cdist().apply

</code></pre>
Pytorch Open Source

Ibrahim_Banat: <p>Hello,<br>
Can you tell me how to master Pytorch and start contributing to the open source documentation, and when ti know I have a good knowledge to start contributing ?</p>

albanD: <p>Hi,</p>
<p>You can find some info in the contributing page on github: <a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md">https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md</a><br>
Otherwise I think you can</p>
<ul>
<li>Try and answer people’s answers here by diving into the codebase to understand why it happens</li>
<li>Look for easy issues on github and ask there for guidance on how they should be fixed (for example with the “good first issue” tag or “small”)</li>
</ul>

ptrblck: <p>Adding to <a class="mention" href="/u/alband">@albanD</a>’s good answer:<br>
Don’t think you need to “master” PyTorch before contributing to it.<br>
Try to pick an easy task at the beginning and you’ll learn a lot along the way. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Fastest way to determine backwards compatibility of an operation?

Sam_Shleifer: <p>I frequently run into the problem of figuring out which versions of torch some function or layer like <code>nn.tanh</code> or  <code>torch.bool</code> was introduced in. My current way of doing it is guessing and then searching through <a href="https://github.com/pytorch/pytorch/releases" rel="nofollow noopener">release notes</a> on github. Is there a faster way?</p>

ptrblck: <p><code>git blame</code> might be another option to see the last PR(s) for a method or searching the docs for each release version.<br>
May I ask what your use case is to look for the first release version?</p>

mohaimen: <p>Adding to <a class="mention" href="/u/ptrblck">@ptrblck</a>, you can go to the source code in git hub and then click on blame button and then scroll down to the implementation of the functionality and you should see the history from its creation. I can’t think of any better way than this.</p>

Sam_Shleifer: <p><strong>Use Case:</strong> Reviewing code for the <a href="https://github.com/huggingface/transformers" rel="noopener nofollow ugc">transformers</a>, where we are trying to maintain backwards compatibility with version &gt;= 1.0.0, but our CI tests run in 1.4, so we have to manually check against breaking the backwards compatibility promise.</p>
<p>The git blame idea inspired me to write a script that you can add to your .bashrc/.zshrc/.bash_profile:</p>
<pre data-code-wrap="bash"><code class="lang-bash">_checkout_grep() {  # helper fn
	git checkout $1 &gt; /dev/null 2&gt;&amp;1  # surpress Previous HEAD position msg
	git grep $2 | wc -l
}

check_torch_compat () {
	# check the pytorch compatibility of a function
	# example usage check_torch_compat torch.bool, TODO: For loop
	cd ~/pytorch/docs
	echo "1.0"
	_checkout_grep v1.0.0 $1
	echo "1.1"
	_checkout_grep v1.1.0 $1
	echo "1.2"
	_checkout_grep v1.2.0 $1
	echo "1.3"
	_checkout_grep v1.3.0 $1
	echo "1.4"
	_checkout_grep v1.4.0 $1
	echo "master"
	_checkout_grep master $1
	cd -  &gt; /dev/null 2&gt;&amp;1
}

</code></pre>
<p>Output for <code>check_torch_compat torch.bool</code> is below. (you have to clone the repo and put it at ~ or update the bash code with your repo path).<br>
(0 in output means <code>torch.bool</code> is not in that version)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4b29ea83de77f2afc6b7500a77219d9659b58948.png" alt="image" data-base62-sha1="aIVC8hTkwNyONAJTz3NKLd59eVq" width="654" height="333"></p>
<p>The best non-scripted solution I’ve heard is manipulating the URL in <a href="https://pytorch.org/docs/1.0.0/nn.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn — PyTorch master documentation</a> + cmd-f. for the function name.</p>

ptrblck: <p>That would be one approach, but I think it might be a flaky one as your CI seems to be only checking for the definition of these functions.<br>
While you could <em>probably</em> make sure that the code is at least executable, you cannot give any guarantee that it’s indeed working.</p>
Uniform class batch training vs. Not uniform class batch training

94Andrew: <p>When you train deep learning model with SGD, batch composes of randomly selected data or sequentially selected data as a default.<br>
I want to compare between uniform class batch training and not uniform class batch training.<br>
More specifically, for CIFAR-10 task, when batch size = 10, each batch has 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 labeled data. Otherwise, Not uniform class batch training means that each batch has 0, 0, 0, 2, 4, 5, 5, 7, 8, 9. You don’t care the order of labels.</p>

ptrblck: <p>For the first approach, you could access the targets in the CIFAR10 dataset and store the data indices for each label class.<br>
Then inside a custom <code>Dataset</code>, you could load a sample for each class and return the complete batch in the <code>__getitem__</code> method.<br>
Note that for this approach you would manually create the batch in <code>Dataset.__getitem__</code>, thus you would have to set <code>batch_size=1</code> in your <code>DataLoader</code>.</p>
<p>If I understand the second use case correctly, you are just randomly sampling the data, so setting <code>shuffle=True</code> in your <code>DataLoader</code> should work.</p>
Changing quant_max and quant_min doesn't have any effect

mg1371: <p>I’m trying to have a more limited range for values in my model. I set quant_max and quant_min to the new values as an argument in FakeQuantize, but when I actually print int_repr() of values, it’s still between 0-255 (or -128 to 127).</p>

jerryzh168: <p>right now the observer uses fixed range depending on dtype: <a href="https://github.com/pytorch/pytorch/blob/master/torch/quantization/observer.py#L153-L162" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/quantization/observer.py#L153-L162</a>, feel free to add the support for quant_min and quant_max for observer.<br>
cc <a class="mention" href="/u/raghuramank100">@raghuramank100</a> for sub 8 bit observer support.</p>

raghuramank100: <p>Yes, we have a PR in the works for supporting sub 8 bit quantization at: <a href="https://github.com/pytorch/pytorch/pull/33743" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/33743</a>, expect to land this end of next week</p>
Distributed ImageNet Example Multi GPU Question

ayushm-agrawal: <p>Hello,<br>
In the ImageNet example linked here <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" rel="nofollow noopener">https://github.com/pytorch/examples/blob/master/imagenet/main.py</a>, when we call the main_worker through mp.spawn, how is the main_worker getting the GPU argument? When I try to run this with 2 nodes that have 2 GPUs each, I always see this parameter to be None. It works for multiple nodes with single GPUs.</p>

mrshenli: <p><code>multiprocessing.spawn</code> will feed the process id as the first argument to the target function. Here is the API doc: <a href="https://pytorch.org/docs/stable/multiprocessing.html#torch.multiprocessing.spawn" rel="nofollow noopener">https://pytorch.org/docs/stable/multiprocessing.html#torch.multiprocessing.spawn</a></p>
How to use Bilinear interpolation to reduce the number of feature maps?

tideng777: <p>I am a freshman and I meet an issue.</p>
<p>I have a batch of feature maps [640, 28, 28]. I want to use bilinear interpolation to convert to  [3, 28, 28]. How can I do that?</p>
<p>The function torchvision.transforms.Resize looks like do not have this option.</p>

ptrblck: <p>Interpolation techniques, such as <code>F.interpolate</code>, work usually in the spatial dimensions, while it seems you would like to reduce the number of channels.<br>
If that’s the case, you could use a conv layer with <code>in_channels=640</code>, <code>out_channels=3</code> and <code>kernel_size=1</code>.</p>
<p>Alternatively you could also permute the input and use <code>F.interpolate</code> on the original channel dimension, which would be then in a spatial dim.</p>

tideng777: <p>Thanks, I got it!!! <img src="https://discuss.pytorch.org/images/emoji/apple/muscle.png?v=9" title=":muscle:" class="emoji" alt=":muscle:"> <img src="https://discuss.pytorch.org/images/emoji/apple/muscle.png?v=9" title=":muscle:" class="emoji" alt=":muscle:"> <img src="https://discuss.pytorch.org/images/emoji/apple/muscle.png?v=9" title=":muscle:" class="emoji" alt=":muscle:"> <img src="https://discuss.pytorch.org/images/emoji/apple/muscle.png?v=9" title=":muscle:" class="emoji" alt=":muscle:"></p>
How to predict set of words given an unordered set

etchner: <p>Hello,</p>
<p>I’m trying to build a Neural Network to, given a set of keywords extracted from a papers dataset, predict another set of keywords extracted from another dataset. For example:</p>
<p>cat, dog, bird -&gt; Mosquito, elephant, Table<br>
Plane -&gt; computer, Pencil<br>
Tablet, house, France -&gt; Italy</p>
<p>As you can see, the input lenght can be different on each set of words, and the output target size can vary too.</p>
<p>Firstly, I tried with a very simple model with three linear layers and the results were very (very) poor. After reading some tutorials, I tried to create a LSTM but that type of Neural Network is not suitable for my problem because the order of the keywords is not relevant for me, I’m not trying to predict sequences.</p>
<p>Finally, I tried adapting the Word Embedding model from this tutorial <a href="https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html</a> but results are very poor and, in addition, I don’t know how to deal with the different length of the input targets (in the tutorial, all the input consists always in two words).</p>
<p>So, I’m a bit lost and I don’t know if a Neural Network if really what I need to solve my problem.</p>
<p>Thanks</p>
Multiply each element with a column from a matrix

RavenTheCute: <p>Suppose I have a matrix <strong>B</strong>, where its elements are <em>b_ij</em>(i-th row, j-th column).<br>
I have another matrix <strong>U</strong>, where its columns are <em>u_j</em>(j-th column).</p>
<p>How can I multiply each <em>b_ij</em> with respect to column <em>u_j</em> in <strong>U</strong>, then sum up all the results in each row of <strong>B</strong>? (that is, $ \sum b_{ij}u_j $ for each i)</p>
<p>Thanks.</p>

abodh_ltd: <p>Let me see if I get your problem right: are you talking something like this?</p>
<pre><code class="lang-python">b = b.view(1,-1)
for i in range(col_u):
       sum = 0
       for j in range(length(b)): # iterating through cols in u
               sum += b[j]* u[: , i]

        A[i,:] = sum


      

</code></pre>

RavenTheCute: <p>I am so sorry. I had some typo in the previous version. <strong>A</strong> is a typo, I am trying to sum up the result for each row of <strong>B</strong>. Thanks for your response <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

RavenTheCute: <p>Yes, I think you got it. Except for the j, I think it should be in range(col_B * i+1, col_B * (i+1) + 1)</p>
All_reduce a list of tensors

babababadukeduke: <p>I am getting the following error when I am trying to all reduce a list of tensor</p>
<p><code>RuntimeError: Tensors must be contiguous</code></p>
<p>Here is a snippet of code</p>
<pre><code class="lang-auto">t_ = [...] # list of tensors
for t in t_: 
   dist.all_reduce(t, , op=dist.ReduceOp.SUM, group=group)
</code></pre>

babababadukeduke: <p>Figure it out! I have to make the tensor contiguous before reducing it. This <a href="https://discuss.pytorch.org/t/contigious-vs-non-contigious-tensor/30107/2">article</a> provides a nice description of it.</p>
[quantization] how to quantize model which include not support to quantize layer

zhangyongwei93: <p>Hi, I have a model which include <code>prelu</code> layer, not support to quantize in current pytorch version, how to  quantize this model for x86 CPU now?  I try to define this model with the following format:</p>
<pre><code class="lang-python">self.convbn1 = QuantizableConvBNBlock(xxx) (has defined)
self.prelu = nn.PReLU()
self.convbn2 = QuantizableConvBNBlock(xxx)        
self.quant = torch.quantization.QuantStub()
self.dequant = torch.quantization.DeQuantStub()
        
def forward(self, x):
    x = self.quant(x)
    x = self.convbn1(x) 
    x = self.dequant(x)
    x = self.prelu(x)
    x = self.quant(x)
    x = self.convbn2(x)
    ...
</code></pre>
<p>but I found after perform the quantization-aware training following tutorial, eval result is very terrible, what is the reason and how to solve it ?<br>
Thanks</p>

jerryzh168: <p>you’ll need to use different quant stub module for each usage because it is recording tensor statistics at that point.<br>
e.g.</p>
<pre><code class="lang-python">x = self.quant1(x)
...
x = self.dequant(x)
...
x = self.quant2(x)
...
x = self.dequant(x)
</code></pre>
Need clarification regarding "register_backward_hook" function

csailnadi: <p>I could not find a good documentation on <code>register_backward_hook()</code> function.</p>
<p>I cannot figure out what it receives for the argument <code>grad_in</code>.</p>
<pre><code class="lang-auto">
    def hook_layers(self):
        def hook_function(module, grad_in, grad_out):
            self.gradients = grad_in[0]
            print("len(grad_in) ", len(grad_in))
            print("grad_in[0].shape ", grad_in[0].shape)
            print("grad_in[1].shape ", grad_in[1].shape)
            print("grad_in[2].shape ", grad_in[2].shape)

        # Register hook to the first layer
        first_layer =list(self.model.features._modules.items())[0][1]
        first_layer.register_backward_hook(hook_function)
</code></pre>
<p>For ImageNet for the first layer I get the following print output:</p>
<pre><code class="lang-auto">len(grad_in) 3
grad_in[0].shape torch.Size([1, 3, 224, 224])
grad_in[1].shape torch.Size([64, 3, 11, 11])
grad_in[2].shape torch.Size([64])
</code></pre>
<p>I do not quite get what those 3 tuple dimensions correspond to.</p>
<p>When I run the same on VGG16 network I get:</p>
<pre><code class="lang-auto">len(grad_in) 3
grad_in[0]  None
grad_in[1].shape torch.Size([64, 3, 3, 3])
grad_in[2].shape torch.Size([64])
</code></pre>
<p>Here the first tuple item is None.</p>
<p>The way it is used:</p>
<pre><code class="lang-auto"> def generate_gradients(self, input_image, target_class):
        model_output = self.model(input_image)
        self.model.zero_grad()
        one_hot_output = torch.FloatTensor(1, model_output.size()[-1]).zero_()
        one_hot_output[0][target_class] = 1
        model_output.backward(gradient=one_hot_output)
</code></pre>

albanD: <p>Hi,</p>
<p>You can check the doc for it <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Module.register_backward_hook">here</a> in particular the warning in there that explains that this is a known limitation (or bug) of the current implementation.</p>

csailnadi: <p>oh, yeah I missed the warning and was very puzzled about the behavior.</p>
Conv2D input and output shape

kbtorcher: <p>I have created a variable <code>conv1 = nn.Conv2d</code> with in_channels = 256, out_channels = 3, kernel_size = 3, and stride = 1. Other method arguments are set by default. I would like to get the input and output shape of this convolution. I tried using conv1.shape and conv1.size() but the error thrown back says no such attribute found. While I can take calculate it by the formula given in the documentation, I would like to directly call a function. Is this possible?</p>

simaiden: <p>Just create a dummy input and pass into the conv layer and check the output size</p>

abodh_ltd: <p>Can you pass in an arbitrary size input first and post the result here? We might be able to you then if you don’t get it.</p>
Indexing using the results returned by sorted

laoreja: <p>Hi,<br>
I have two tensors of the same shape: (batch_size, nchannels, n),<br>
I would like to sort one of the tensor, and then reorder the other tensor using the indices returned by sort.</p>
<p>Because currently, index_select only support vector, so I use some work around to make my stuff work:</p>
<pre><code class="lang-auto">a = torch.randn(2,3,4)
sorted_a, indices = a.sort(dim=1)

new_indices = indices * a.size(-1)
batch_add_on = torch.arange(0, a.size(0))[:, None, None].repeat(1, a.size(1), a.size(2)).long() * a.size(1) * a.size(2)
last_dim_add_on = torch.arange(0, a.size(-1))[None, None, :].repeat(a.size(0), a.size(1), 1).long()
new_indices = new_indices + batch_add_on + last_dim_add_on

b = a
sorted_b = torch.index_select(b.view(-1), dim=0, index=new_indices.view(-1)).view(a.size(0), a.size(1), a.size(2))
</code></pre>
<p>But in this way, when training using multiple GPUs, such index method would cause some problems, if I understand data parallelism correctly?</p>
<p>Do you have a better way to do such indexing, or is data parallelism smart enough?</p>

Alberto_Gomez: <p>Did you find a better solution?</p>
What is the correct LSTM time-series input order?

ruzbehakbar: <p>Assuming we have a Sequence-to-Sequence LSTM model for time-series prediction:</p>
<p>Input time-series:  X shaped as (batch_size, seq_length = N, input_dim = 1)<br>
Output time-series: y shaped as (batch_size, seq_length = N, input_dim = 1)</p>
<p>I want to predict time series of y using N-lagged X data.  What is the correct order (for preprocessing) of the input data into the LSTM mode. In other words, in what direction are the data fed into LSTM models?</p>
<p>Should the input tensor/array, X, be ordered as:</p>
<p>Option 1:</p>
<p>input slice (X[0,:])   [x(t), x(t-1),…, x(t-1)]<br>
output:                    [y(t), y(t-1),…, y(t-1)]</p>
<p>e.g. model( [x(t), x(t-1),…, x(t-1)])</p>
<p>or, Option 2:</p>
<p>input slice (X[0,:])   [x(t-N),…, x(t-1), x(t)]<br>
output:                    [y(t-N),…, y(t-1), y(t)]</p>
<p>e.g., model([x(t-N),…, x(t-1), x(t)])</p>
<p>Basically, for my input data, does the sequence time-step go from (t) to (t-N) left to right, or vise vera?</p>

sharvil: <p>Your sequence should be ordered <code>[x(t-N) ... x(t-1), x(t)]</code>. Keep in mind that the sequence dimension should be first unless you specify <code>batch_first=True</code> on the LSTM layer. Your X and Y variables should be shaped <code>[T,N,C]</code> where <code>T</code> is the sequence length, <code>N</code>, is the batch size, and <code>C</code> is the feature size.</p>

ruzbehakbar: <p>Thanks!</p>
<p>Just to clarify, if input data shape is [T,N,C] then <code>batch_first = False</code>, and if [N,T,C] , then<code> batch_first = True</code></p>
<p>T = Seq. length.<br>
N = batch size<br>
C = # of features.</p>

sharvil: <p>Yup, that’s right.<br>
…</p>
Multiply 2D tensor with 3D tensor

danielfllaneza: <p>I have a 2D tensor [64,149] were 64 is the batch size and a tensor [64,149,128] (64 is the batch size, too). I would like to get the tensor [64,128]. torch.matmul has not worked so far as it provides me torch.Size([64, 64, 128]) as a result and torch.bmm throws the following error:</p>
<pre><code class="lang-auto">*** RuntimeError: invalid argument 6: expected 3D tensor at /opt/conda/conda-bld/pytorch_1570910687230/work/aten/src/THC/generic/THCTensorMathBlas.cu:518
</code></pre>
Effects of CUDA out of memory on shared GPU

Andrea_Rosasco: <p>I was given access to a remote workstations where I can use a GPU to train my model. Of course all the resources are shared and the GPU memory is often partially used by other people processes.<br>
Does getting a CUDA out of memory error means I just made someone else’s process crash?</p>

albanD: <p>It’s possible but not sure.<br>
Basically whichever user asks for memory when it is full will see its process getting killed.<br>
So it can happen to only one user, or all of them if they all happen to ask for memory at that time.</p>

Andrea_Rosasco: <p>Thanks for the answer! Anyway, that’s too bad… I don’t want to kill someone else’s model that may be training for a lot and I don’t want my model killed for the same reason! Have you ever worked in such scenario? Do you know of any tricks to prevent it?</p>

ptrblck: <p>I don’t think there is a good solution. Even if you would limit your code to e.g. 25% of the device memory, other processes might still want to use the whole GPU and thus creating an OOM error. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
Error in size empy

Peyman: <p>hi<br>
when i am euning in code, i have error<br>
for cnt in range(_batch_size):<br>
tmp_data = dataset_info[data_idx[cnt]]</p>
<h1>print(tmp_data)</h1>
<h1>if len(_image.shape)==3:</h1>
<h1>_image = cv2.cvtColor(_image, cv2.COLOR_BGR2RGB)</h1>
<h1>plt.imshow(_image)</h1>
<h1>plt.pause(0.001)</h1>
<pre><code>    _rect = tmp_data['rect']
    _rect0=min(0,_rect[0])
    _rect1=min(0,_rect[1])
    _rect2=min(10,_rect[2])
    _rect3=min(10,_rect[3])
            _image = cv2.imread(tmp_data['img_path'])
    print(tmp_data['img_path'])
    print(tmp_data['rect'])
    if _image is not None:            
        cropped_image = _image[_rect1:(_rect1+_rect3), _rect0:(_rect0+_rect2), :]
        if cropped_image is not None:
            if (len(cropped_image.shape)==3):
                cropped_image = cv2.resize(cropped_image, IMG_SIZE)
                
                _lbp = my_lbp_func(cropped_image)
                _new_image = np.concatenate([(cropped_image/255.0),_lbp],axis=2)
                y_Face[cnt] = 1
            else:
                y_Face[cnt] = 0
        else:
            y_Face[cnt] = 0
    else:
        y_Face[cnt] = 0
</code></pre>
<hr>
<p>File “E:\Final\MyDataLoader.py”, line 161, in load_data_for_net<br>
cropped_image = cv2.resize(cropped_image, IMG_SIZE)</p>
<p>error: OpenCV(4.2.0) C:\projects\opencv-python\opencv\modules\imgproc\src\resize.cpp:4045: error: (-215:Assertion failed) !ssize.empty() in function ‘cv::resize’</p>

ptrblck: <p>It seems OpenCV is raising this error, as you might have passed an empty numpy array to <code>cv2.resize</code>.<br>
Could you print the shape of <code>cropped_image</code> before passing it to <code>cv2.resize</code>, please?</p>
Batch size suddenly reduced

maroov: <p>Hello, something weird is happening when I try to train my small network. The output size after processing a batch is suddenly reduced to 12 (it is supposed to be 16) at batch <span class="hashtag">#1518</span> of the first epoch.</p>
<pre><code class="lang-auto">RuntimeError: The size of tensor a (16) must match the size of tensor b (12) at non-singleton dimension 0
</code></pre>
<p>This error is produced when computing the loss. But it works perfectly fine for previous batches, which should be all identical (size 16)!</p>
<p>This is even more annoying because every time I execute and get this I have to restart the whole Colab thing, otherwise I get tha nnoyingly vague <code>RuntimeError: CUDA error: device-side assert triggered</code> whichs ‘locks’ the notebook.</p>
<p>This is the code for the training:</p>
<pre><code class="lang-auto">mod1 = Module1().to(device)
optimizer = optim.Adam(mod1.parameters(), lr=lr)

for epoch in trange(epochs):
  for i, data in enumerate(trainloader):
    xt = data[0][0].to(device)  # image patch
    xtk = data[0][1].to(device)  # 'predicted' patch
    optimizer.zero_grad()

    zt = mod1(xt)
    ztk = mod1(xtk)     
    
    loss = infoNCE(zt, ztk)  # custom loss function
    loss.backward()
    optimizer.step()
</code></pre>
<p>My current network:</p>
<pre><code class="lang-auto">class Module1(nn.Module):

  def __init__(self):
    super(Module1, self).__init__()
    self.conv1 = nn.Conv2d(1, 32, 5, 1, 2)
    self.conv2 = nn.Conv2d(32, 64, 5, 1, 2)
    self.conv3 = nn.Conv2d(64, 128, 5, 1, 2)
    self.conv4 = nn.Conv2d(128, 128, 5, 1, 2)

    self.pool = nn.MaxPool2d(2, 2)
    self.fc = nn.Linear(128*1*1, 3)  # 3 features (NORB)

  def forward(self, x):
    x = self.pool(F.relu(self.conv2(F.relu(self.conv1(x)))))
    x = self.pool(F.relu(self.conv4(F.relu(self.conv3(x)))))
    x = self.pool(self.pool(x))
    x = x.view(x.size(0), -1)
    x = self.fc(x)
    return x
</code></pre>
<p>My custom loss:</p>
<pre><code class="lang-auto">def infoNCE(zt, ztk):
  ind = np.diag_indices(b_size)  # batch size 16
  mul = ztk @ torch.t(zt)
  num = mul[ind[0], ind[1]]
  m, _ = torch.max(mul, axis=0)
  den = torch.log(torch.sum(torch.exp(mul - m), axis=0)) + m
  val = torch.mean(-num + den)
  return val
</code></pre>
<p>Note that everything was working for a different dataset (CIFAR-10) but now it breaks (NORB). The inputs for the network should have the same shape though, so I don’t know what is wrong.</p>

JuanFMontesinos: <p>In the data loader ser drop_last=True<br>
If n_samples%batch_size!=0 your last batch will be smaller</p>

maroov: <p>You’re right, that was the issue. Thank you!</p>
Update a tensor at the intersection of row and column indices

sisaman: <p>I have a 2D bool tensor <code>mask</code> and I would like to update its values at the intersection of the given row and col indices, like the following:</p>
<pre><code class="lang-auto">mask = torch.zeros(5,3, dtype=torch.bool)

rows = torch.tensor([0,2,4]) 
cols = torch.tensor([1,2])

mask[rows, cols] = True
</code></pre>
<p>But I receive the following error:</p>
<pre><code class="lang-auto">IndexError: shape mismatch: indexing tensors could not be broadcast together with shapes [3], [2]
</code></pre>
<p>The desired output is:</p>
<pre><code class="lang-auto">tensor([[False, True,  True ],
        [False, False, False],
        [False, True,  True ],
        [False, False, False],
        [False, True,  True ]])
</code></pre>
<p>How is that possible in PyTorch?</p>

sisaman: <p>My question has been answered here:</p>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/61117634/how-to-change-the-values-of-a-2d-tensor-in-certain-rows-and-columns" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/2571755/sisaman" target="_blank" rel="nofollow noopener">
    <img alt="sisaman" src="https://i.stack.imgur.com/DLmg2.jpg?s=128&amp;g=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/61117634/how-to-change-the-values-of-a-2d-tensor-in-certain-rows-and-columns" target="_blank" rel="nofollow noopener">How to change the values of a 2d tensor in certain rows and columns</a>
</h4>

<div class="tags">
  <strong>python, pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/2571755/sisaman" target="_blank" rel="nofollow noopener">
    sisaman
  </a>
  on <a href="https://stackoverflow.com/questions/61117634/how-to-change-the-values-of-a-2d-tensor-in-certain-rows-and-columns" target="_blank" rel="nofollow noopener">09:08AM - 09 Apr 20 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Using LSTM predictions as inputs

DrunkAlex: <p>Hello all,</p>
<p>I am working with LSTM for multivariate time series prediction. Let’s imagine the situation: I have 2 time series, A and B, and I want to predict t-value of B using previous values of A and B (before t). Such prediction works fine, my model gets good result.<br>
But what if (during testing, after training) I want to use predicted values of B as inputs for next time step instead of real values? For example: I predict first value of B, make a step, put predicted value instead of a real, and make prediction again. Then I use two predicted values instead of real two, and so on.</p>
<p>Are there any possibilities to do that?</p>
My loss is showing random behavior

Tushar_Vatsa: <p>for steps,(x,y) in enumerate(loader):</p>
<pre><code>    count=count+1

    if net.train(mode=True):
        logits= net(x.float())
        target=y.long()
        loss=criterion(logits,target)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
    else:
        logits= net(x.float())
        target=y.long()
        loss=criterion(logits,target)
</code></pre>
<p>I have not used softmax as I am calling cross entropy.<br>
Forward Propagation:<br>
x=self.act(self.layer1(x))<br>
x=self.layer2(x)<br>
Here is my output:<br>
tensor(1.8975, grad_fn=) Loss of Training Data<br>
tensor(1.9753, grad_fn=) loss of test set<br>
tensor(1.7179, grad_fn=) Loss of Training Data<br>
tensor(1.5440, grad_fn=) loss of test set<br>
tensor(1.6857, grad_fn=) Loss of Training Data<br>
tensor(1.4142, grad_fn=) loss of test set<br>
tensor(1.3730, grad_fn=) Loss of Training Data<br>
tensor(2.0773, grad_fn=) loss of test set<br>
tensor(1.5744, grad_fn=) Loss of Training Data<br>
tensor(1.9398, grad_fn=) loss of test set<br>
tensor(1.3640, grad_fn=) Loss of Training Data<br>
tensor(1.0328, grad_fn=) loss of test set<br>
tensor(1.3167, grad_fn=) Loss of Training Data<br>
tensor(1.5097, grad_fn=) loss of test set<br>
tensor(1.6103, grad_fn=) Loss of Training Data<br>
tensor(1.2433, grad_fn=) loss of test set<br>
tensor(1.7091, grad_fn=) Loss of Training Data<br>
tensor(1.3714, grad_fn=) loss of test set<br>
tensor(1.2498, grad_fn=) Loss of Training Data<br>
tensor(1.6427, grad_fn=) loss of test set<br>
tensor(1.8755, grad_fn=) Loss of Training Data<br>
tensor(1.3751, grad_fn=) loss of test set<br>
tensor(1.2940, grad_fn=) Loss of Training Data<br>
tensor(1.6323, grad_fn=) loss of test set<br>
tensor(1.4709, grad_fn=) Loss of Training Data<br>
tensor(1.2648, grad_fn=) loss of test set<br>
tensor(1.3212, grad_fn=) Loss of Training Data<br>
tensor(1.7257, grad_fn=) loss of test set<br>
tensor(1.4292, grad_fn=) Loss of Training Data<br>
tensor(1.9744, grad_fn=) loss of test set<br>
tensor(1.4333, grad_fn=) Loss of Training Data<br>
tensor(1.4438, grad_fn=) loss of test set<br>
tensor(1.3763, grad_fn=) Loss of Training Data<br>
tensor(1.5208, grad_fn=) loss of test set<br>
tensor(1.4129, grad_fn=) Loss of Training Data<br>
tensor(1.6961, grad_fn=) loss of test set<br>
tensor(1.2673, grad_fn=) Loss of Training Data<br>
tensor(1.3677, grad_fn=) loss of test set<br>
tensor(1.4119, grad_fn=) Loss of Training Data<br>
tensor(1.5740, grad_fn=) loss of test set</p>
LibTorch on tvOS?

ar97: <p>Hi,</p>
<p>Is it possible to use LibTorch on tvOS platform? The <code>pod install</code> command gives following error message -</p>
<blockquote>
<p>[!] The platform of the target <code>HelloWorld</code> (tvOS 13.2) is not compatible with <code>LibTorch (1.4.0)</code>, which does not support <code>tvOS</code>.</p>
</blockquote>
FLOPS in Conv2d and ConvTransposed2d

lepoeme20: <p>Hi,</p>
<p>After calculating the FLOPS of the model (GAN), I found a strange point.</p>
<p>I compared following two cases:</p>
<pre><code class="lang-auto">1. Conv2d(kernel_size=3, stride=1, padding=1)
2. ConvTranspose2d(kernel_size=4, stride=2, padding=1)
</code></pre>
<p>The dimensions of the input channel and output channel are the same as 1024 and 512 respectively.</p>
<p>And my computation results are:</p>
<pre><code class="lang-auto">Conv2d
    - # of parameters: 4.719 M
    - GMac: 3.7

ConvTransposed2d
    - # of parameters: 8.389 M
    - GMac: 1.645
</code></pre>
<p>First, I would like to ask if the above results are valid.<br>
If so, I’d like to ask why Conv2d’s FLOPS with a small number of parameters is larger than ConvTransposed2d.</p>

ptrblck: <p>The number of parameters seems to be correct and you could simply verify it via:</p>
<pre><code class="lang-python">nb_params = conv.weight.nelement() # + conv.bias.nelement() (you can skip the bias, as it's insignificant compared to the weights)
</code></pre>
<p>Since your transposed convolution uses a stride of 2, the kernels will be applied to less patches than the conv layer. However, each kernel is larger, which will increase the number of theoretical flops again.</p>
Pytorch cuda speed

jack_ma: <p>Hi, I’ve read this tutorial <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/cpp_extension.html</a>.<br>
But I feel weird about the speed result.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/e/bee2a312ddb10310f16cf5c6553971e9b7857cbc.png" data-download-href="https://discuss.pytorch.org/uploads/default/bee2a312ddb10310f16cf5c6553971e9b7857cbc" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/bee2a312ddb10310f16cf5c6553971e9b7857cbc_2_690x266.png" alt="image" data-base62-sha1="reEhMY4qbCE5PDcku99DC8kSyzG" width="690" height="266" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/bee2a312ddb10310f16cf5c6553971e9b7857cbc_2_690x266.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/bee2a312ddb10310f16cf5c6553971e9b7857cbc_2_1035x399.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/e/bee2a312ddb10310f16cf5c6553971e9b7857cbc.png 2x" data-dominant-color="F6F7F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1176×454 27.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
It says CUDA-based C++ code is slower than custom CUDA kernel. The two method both run codes on cuda device in c++ with the same operator. Why there is a speed gap between them?<br>
Thank you.</p>

ptrblck: <p>As described in the tutorial, the next step after writing the extension is to write a custom CUDA kernel and reduce the kernel launch overheads:</p>
<blockquote>
<p>A definite method of speeding things up is therefore to rewrite parts in C++ (or CUDA) and  <em>fuse</em>  particular groups of operations. Fusing means combining the implementations of many functions into a single functions, which profits from fewer kernel launches as well as other optimizations we can perform with increased visibility of the global flow of data.</p>
</blockquote>
<blockquote>
<p>For the LLTM, this has the prospect of being particularly effective, as there are a large number of pointwise operations in sequence, that can all be fused and parallelized in a single CUDA kernel.</p>
</blockquote>