How to acclerate my c++ cuda code?

username: flyerjia
cooked: <p>My c++ cuda code is slow. What can i do to acclerate my code?</p>

username: tom
cooked: <p>I don’t think this is answerable without more detail.</p>
<p>Best regards</p>
<p>Thomas</p>

username: flyerjia
cooked: <p>Hi, thanks,my code:</p>
<pre><code class="lang-auto">std::vector&lt;torch::Tensor&gt; outputlayer_cuda_backward(torch::Tensor input, torch::Tensor grad_indexs, torch::Tensor grad_values, torch::Tensor batch_indexs, torch::Tensor weight){
    const int input_dimension = input.size(1);
    const int batch_size = input.size(0);
    const int grad_size = grad_indexs.size(1);
    const int threads = 1024;
    const dim3 blocks((input_dimension + threads -1)/threads, batch_size);
    auto grad_input = torch::zeros_like(input);
    AT_DISPATCH_FLOATING_TYPES(grad_values.type(), "outputlayer input backward", ([&amp;]{
        outputlayer_cuda_input_backward_kernel&lt;scalar_t&gt;&lt;&lt;&lt;blocks, threads&gt;&gt;&gt;(
            grad_indexs.packed_accessor32&lt;int32_t, 2, torch::RestrictPtrTraits&gt;(),
            grad_values.packed_accessor32&lt;scalar_t, 1, torch::RestrictPtrTraits&gt;(),
            batch_indexs.packed_accessor32&lt;int32_t, 2, torch::RestrictPtrTraits&gt;(),
            weight.packed_accessor32&lt;scalar_t, 2, torch::RestrictPtrTraits&gt;(),
            grad_input.packed_accessor32&lt;scalar_t, 2, torch::RestrictPtrTraits&gt;(),
            input_dimension,
            batch_size,
            grad_size
        );
    }));
    auto grad_weight = torch::zeros_like(weight);
    const int blocks1 = (grad_size + threads - 1)/threads;
    AT_DISPATCH_FLOATING_TYPES(grad_values.type(), "outputlayer weight backward", ([&amp;]{
        outputlayer_cuda_weight_backward_kernel&lt;scalar_t&gt;&lt;&lt;&lt;blocks1, threads&gt;&gt;&gt;(
            input.packed_accessor32&lt;scalar_t, 2, torch::RestrictPtrTraits&gt;(),
            grad_indexs.packed_accessor32&lt;int32_t, 2, torch::RestrictPtrTraits&gt;(),
            grad_values.packed_accessor32&lt;scalar_t, 1, torch::RestrictPtrTraits&gt;(),
            grad_weight.packed_accessor32&lt;scalar_t, 2, torch::RestrictPtrTraits&gt;(),
            input_dimension,
            grad_size
        );
    }));
    auto grad_bias = torch::zeros({weight.size(1)}, at::device(at::kCUDA));
    AT_DISPATCH_FLOATING_TYPES(grad_values.type(), "outputlayer bias backward", ([&amp;]{
        outputlayer_cuda_bias_backward_kernel&lt;scalar_t&gt;&lt;&lt;&lt;blocks1, threads&gt;&gt;&gt;(
            grad_indexs.packed_accessor32&lt;int32_t, 2, torch::RestrictPtrTraits&gt;(),
            grad_values.packed_accessor32&lt;scalar_t, 1, torch::RestrictPtrTraits&gt;(),
            grad_bias.packed_accessor32&lt;scalar_t, 1, torch::RestrictPtrTraits&gt;(),
            grad_size
        );
    }));
    return {grad_input, grad_weight, grad_bias};
}

__global__ void outputlayer_cuda_input_backward_kernel(
    const torch::PackedTensorAccessor32&lt;int32_t,2,torch::RestrictPtrTraits&gt; grad_indexs,
    const torch::PackedTensorAccessor32&lt;scalar_t,1,torch::RestrictPtrTraits&gt; grad_values,
    const torch::PackedTensorAccessor32&lt;int32_t,2,torch::RestrictPtrTraits&gt; batch_indexs,
    const torch::PackedTensorAccessor32&lt;scalar_t,2,torch::RestrictPtrTraits&gt; weight,
    torch::PackedTensorAccessor32&lt;scalar_t,2,torch::RestrictPtrTraits&gt; grad_input,
    const int32_t input_dimension,
    const int32_t batch_size,
    const int32_t grad_size){
    const int row = blockIdx.y * blockDim.y + threadIdx.y;
    const int col = blockIdx.x * blockDim.x + threadIdx.x;
    if(row &lt; batch_size &amp;&amp; col &lt; input_dimension){
        float sum = 0.0;
        for(int k = batch_indexs[row][0]; k &lt;= batch_indexs[row][1]; k++){
            sum += grad_values[k] * weight[col][grad_indexs[1][k]];
        }
        grad_input[row][col] = sum;
    }

}

__global__ void outputlayer_cuda_weight_backward_kernel(
    const torch::PackedTensorAccessor32&lt;scalar_t,2,torch::RestrictPtrTraits&gt; input,
    const torch::PackedTensorAccessor32&lt;int32_t,2,torch::RestrictPtrTraits&gt; grad_indexs,
    const torch::PackedTensorAccessor32&lt;scalar_t,1,torch::RestrictPtrTraits&gt; grad_values,
    torch::PackedTensorAccessor32&lt;scalar_t,2,torch::RestrictPtrTraits&gt; grad_weight,
    const int32_t input_dimension,
    const int32_t grad_size){
    const int col = blockIdx.x * blockDim.x + threadIdx.x;
    if(col &lt; grad_size){
        for(int k = 0; k &lt; input_dimension; k++){
            grad_weight[k][grad_indexs[1][col]] += grad_values[col] * input[grad_indexs[0][col]][k];
        }
    }
}
__global__ void outputlayer_cuda_bias_backward_kernel(
    const torch::PackedTensorAccessor32&lt;int32_t,2,torch::RestrictPtrTraits&gt; grad_indexs,
    const torch::PackedTensorAccessor32&lt;scalar_t,1,torch::RestrictPtrTraits&gt; grad_values,
    torch::PackedTensorAccessor32&lt;scalar_t,1,torch::RestrictPtrTraits&gt; grad_bias,
    const int32_t grad_size){
    const int col = blockIdx.x * blockDim.x + threadIdx.x;
    if(col &lt; grad_size){
        grad_bias[grad_indexs[1][col]] += grad_values[col];
    }
}
</code></pre>
<p>Do you any suggestions?</p>

username: tom
cooked: <p>So why do you think it’s slow?<br>
Some observations regarding <code>outputlayer_cuda_input_backward_kernel</code>:</p>
<ul>
<li>I usually read the upper bounds of my for loops into a local variable. (I’m never sure whether it would be strictly necessary, but you want that to be cached in a register.)</li>
<li>If the k loop is large, you can parallelize the reduction by partially reducing within a thread and then across threads. This is a somewhat common pattern, I’m trying to describe it in <a href="http://lernapparat.de/sinkhorn-kernel/" rel="nofollow noopener">my sinkhorn kernel blog post</a>.</li>
<li>In the for loop assigning <code>weight[col]</code> to a local (<code>auto weight_col = weight[col]</code>) might reduce some indexing.</li>
</ul>
<p>Moving to <code>outputlayer_cuda_weight_backward_kernel</code>:</p>
<ul>
<li>I would cache the indices <code>grad_indexs[1][col]</code> as it is constant in the for loop, for the input, you might also cache the partially indexed accessor (like above).</li>
<li>For <code>grad_weight[k][grad_indexs[1][col]] +=</code> you need to be sure that no two threads will try to write to the same memory location for this to even be correct. If not, this is where the dreaded (for determinism and performance unless accesses are sparse) indexAdd comes into play. You would need to do some relatively elaborate scanning f that doesn’t work.</li>
</ul>
<p>Similar considerations around the <code>+=</code> apply to the bias backward.</p>
<p>I haven’t really looked at the thread/grid layout. You likely want to benchmark things as you try the above for potential improvements.</p>
<p>Best regards</p>
<p>Thomas</p>

username: flyerjia
cooked: <p>Thank you for your replying.<br>
In <code>outputlayer_cuda_weight_backward_kernel</code>,   <code>grad_weight[k][grad_indexs[1][col]] +=</code> can work correctly(the data is very sparse).<br>
I will try above suggestions. Do you know what toolkits can anaylze the arithmetic workload of GPU, which can be used in pytorch?</p>
<p>Best regards<br>
Jia</p>

username: tom
cooked: <aside class="quote no-group" data-username="flyerjia" data-post="5" data-topic="60138">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/cdc98d/48.png" class="avatar"> flyerjia:</div>
<blockquote>
<p>Do you know what toolkits can anaylze the arithmetic workload of GPU, which can be used in pytorch?</p>
</blockquote>
</aside>
<p>I’ve never used anything more fancy than nvvp (though I guess nsight is the more modern thing).<br>
Most kernels I write are memory bandwith bound, so that and the parallelization pattern are the first two things to look at.</p>
<p>Best regards</p>
<p>Thomas</p>