ATen cuda kernel dispatch for type

username: dashesy
cooked: <p>I am using ATen and <a href="https://github.com/pytorch/extension-cpp/blob/master/cuda/lltm_cuda_kernel.cu" rel="nofollow noopener">this</a> example, from <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="nofollow noopener">here</a>.<br>
I have a kernel that needs one array of floats (for input) and one array of ints (for labels), should I still use <code>AT_DISPATCH_FLOATING_TYPES</code>?</p>
<ol>
<li>
<p>does this line mean only floats will be marshalled to cuda kernel? or is it just specifying the scalar_t?</p>
<p>AT_DISPATCH_FLOATING_TYPES(X.type(), “lltm_forward_cuda” …</p>
</li>
<li>
<p>Please correct me if I am wrong: The integral types (int, bool, …) I guess need no special treatment</p>
</li>
</ol>
<p>P.S. it looks like both lines 123 and 157 pass “lltm_forward_cuda” as the second argument, that does not seem right if it is for debugging</p>

username: tom
cooked: <p>Re 1: AT_DISPATCH_FLOATING_TYPES will do float and double automatically.<br>
You specify the dispatch for the type of X. When you have an int-Tensor-argument Y, it’ll be the same. You cannot pass an int-Tensor as X here.<br>
If you want half, there also is a _AND_HALF version.<br>
Re 2: You don’t get any special treatment of these and would have to do it on your own.<br>
Probably using backward in the second string is better.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Jakub_Mitura
cooked: <p>Hello If anybody will be looking for it now there are multpple at dispatch macros here</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/61d6c4386459441710fb4cfa2929a3f77e95e5f7/aten/src/ATen/Dispatch.h">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/61d6c4386459441710fb4cfa2929a3f77e95e5f7/aten/src/ATen/Dispatch.h" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/61d6c4386459441710fb4cfa2929a3f77e95e5f7/aten/src/ATen/Dispatch.h" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/61d6c4386459441710fb4cfa2929a3f77e95e5f7/aten/src/ATen/Dispatch.h</a></h4>


      <pre><code class="lang-h">#pragma once

#include &lt;ATen/core/DeprecatedTypeProperties.h&gt;
#include &lt;c10/macros/Macros.h&gt;
#include &lt;c10/util/Exception.h&gt;
#include &lt;c10/util/Half.h&gt;
#include &lt;c10/util/Metaprogramming.h&gt;
#include &lt;c10/util/complex.h&gt;
#include &lt;c10/util/string_view.h&gt;

#ifdef TEMPLATE_SELECTIVE_BUILD
#include &lt;ATen/selected_mobile_ops.h&gt;
#else
namespace at {
/**
 * The method should_include_kernel_dtype() returns true/false
 * based on whether the switching code for a specific dtype should be
 * included based on build time constants generated from tracing model
 * execution. This method will be implmeneted via code-generation and
 * included in this file when code-gen is ready.
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/61d6c4386459441710fb4cfa2929a3f77e95e5f7/aten/src/ATen/Dispatch.h" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: Max_Unhold
cooked: <p><a class="mention" href="/u/tom">@tom</a>, <a class="mention" href="/u/dashesy">@dashesy</a> Hey, what does _AND_HALF do? So is there still no way to mix tensor types when you give them to the CUDA kernel (long and float)</p>

username: Max_Unhold
cooked: <p><a class="mention" href="/u/jakub_mitura">@Jakub_Mitura</a> Did you figure out, which one is the right one? Any Dispatchers of the form  <code>AT_DISPATCH_CASE_ALL_TYPES_AND_...</code> seem to be candidates.</p>

username: tom
cooked: <p>No, this is not for mixing types, but for quite a while the default floating point types were FP32 and FP64, so _AND_HALF adds FP16 (and maybe BF16?) support.<br>
Given that each dtype produces a separate kernel (with C++ templating) there is not built-in mechanism for mixing two kernels (I did make that manually for CTC if I recall correctly, due to CuDNN wanting 32 bit ints and PyTorch defaulting to 64 bit ints). These really are just a switch statement, so you could easily make your own, but beware of the explosion of the number of kernels.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Max_Unhold
cooked: <p><a class="mention" href="/u/tom">@tom</a> First of all, thank you for your answer. To clarify my statement: I do not want to mix different kernels but just mix <em>the dtypes of the arguments</em>: An example</p>
<p>If all the inputs are of <strong>long</strong> type, the following would work:</p>
<pre><code class="lang-auto">    AT_DISPATCH_INTEGRAL_TYPES(at::ScalarType::Long,"Some long error message", [&amp;] {
      some_cuda_kernel&lt;scalar_t&gt;&lt;&lt;&lt;blocks, threads&gt;&gt;&gt;(
        some_long_tensor.packed_accessor&lt;scalar_t,2,torch::RestrictPtrTraits,size_t&gt;(),
       .... 
      );
      }
    );
</code></pre>
<p>where I have used the packed_accesor.</p>
<p>If I only have <strong>float</strong> tensors, the following works</p>
<pre><code class="lang-auto">  AT_DISPATCH_FLOATING_TYPES(at::ScalarType::Float,"Some floating error message", [&amp;] {
    some_cuda_kernel&lt;scalar_t&gt;&lt;&lt;&lt;blocks, threads&gt;&gt;&gt;(
      some_float_tensor.packed_accessor&lt;scalar_t,2,torch::RestrictPtrTraits,size_t&gt;(),
     .... 
    );
    }
  );
</code></pre>
<hr>
<p>How do I dispatch a kernel that takes both <strong>some_long_tensor</strong> and  <strong>some_float_tensor</strong>?</p>
<p>Attempts with <strong>AT_DISPATCH_ALL_TYPES</strong></p>
<pre><code class="lang-auto">AT_DISPATCH_ALL_TYPES(at::ScalarType,"Hello !",[&amp;] {
    some_cuda_kernel&lt;scalar_t&gt;&lt;&lt;&lt;blocks, threads&gt;&gt;&gt;(
      some_float_tensor.packed_accessor&lt;scalar_t,2,torch::RestrictPtrTraits,size_t&gt;(),
      some_long_tensor.packed_accessor&lt;scalar_t,2,torch::RestrictPtrTraits,size_t&gt;(),
     .... 
    );
})
</code></pre>
<p>or <strong>AT_DISPATCH_INTEGRAL_TYPES_AND</strong></p>
<pre><code class="lang-auto">  AT_DISPATCH_INTEGRAL_TYPES_AND(at::ScalarType::Float,
                                 at::ScalarType::Long,
                                 "some combined error message", [&amp;] {
    some_cuda_kernel&lt;scalar_t&gt;&lt;&lt;&lt;blocks, threads&gt;&gt;&gt;(...)
  }
);
</code></pre>
<p><strong>all failed.</strong></p>
<p>For any raw CUDA kernel, this is not a problem. So if there is any other way to use a raw kernel in this situation, I would appreciate a hint.</p>

username: tom
cooked: <p>You want the first param of <code>packed_accessor</code> (and the PackedAccessor declaration) to be the C++ type you need:</p>
<pre><code class="lang-auto">some_float_tensor.packed_accessor&lt;float,2,...&gt;(),
some_long_tensor.packed_accessor&lt;int64_t, 2...&gt;(),
</code></pre>
<p>The <code>AT_DISPATCH</code> will have a switch querying a <code>at::ScalarType</code> (~the Python dtype)  (of a tensor that might have any of several) and providing the matching C++ type as <code>scalar_t</code> to be used in templating the accessor and kernel call.<br>
If you have fixed dtypes for your arguments, you would not need <code>AT_DISPATCH</code> at all.</p>