Torch.prod produces RuntimeError: CUDA driver error: invalid argument

aliyesilkanat: <p>Hi,</p>
<p>I am experiencing a problem with the <em><strong>prod</strong></em> method.</p>
<pre><code class="lang-auto">torch.Tensor([[32, 32], [16, 16], [8, 8]]).cuda().prod(1)
</code></pre>
<blockquote>
<p>RuntimeError: CUDA driver error: invalid argument</p>
</blockquote>
<p>The driver version is 525.60.13. I tried with two fresh conda environments on python=3.8 by</p>
<blockquote>
<p>conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3 -c pytorch<br>
conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia</p>
</blockquote>
<p>, and got the same error on both.<br>
I also tried setting CUDA_LAUNCH_BLOCKING=1, but the error message didn’t change.</p>
<p>What could be the reason behind this?<br>
Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

arunppsg: <p>Maybe could you try this one: <code>a = torch.Tensor([[32, 32], [16, 16], [8, 8]]).to('cuda').prod(1)</code>?</p>

aliyesilkanat: <p>Didn’t work, same error.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/aliyesilkanat">@aliyesilkanat</a>,</p>
<aside class="quote no-group" data-username="aliyesilkanat" data-post="1" data-topic="179054">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aliyesilkanat/48/44737_2.png" class="avatar"> aliyesilkanat:</div>
<blockquote>
<pre><code class="lang-auto">torch.Tensor([[32, 32], [16, 16], [8, 8]]).cuda().prod(1)
</code></pre>
</blockquote>
</aside>
<p>This works on the latest version of pytorch for me, and returns,</p>
<pre><code class="lang-auto">tensor([1024.,  256.,   64.], device='cuda:0')
</code></pre>
<p>It seems your CUDA install may be broken, can you check your cuda version via <code>torch.version.cuda</code>?</p>

aliyesilkanat: <aside class="quote no-group" data-username="aliyesilkanat" data-post="1" data-topic="179054">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aliyesilkanat/48/44737_2.png" class="avatar"> aliyesilkanat:</div>
<blockquote>
<p><code>torch.Tensor([[32, 32], [16, 16], [8, 8]]).cuda().prod(1)</code></p>
</blockquote>
</aside>
<p>Hi <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a> ,</p>
<p>I created two conda environments related to this issue, and <code>torch.version.cuda</code> s are ‘11.7’ and ‘11.3’. And they both have the same problem.</p>
<p>Edit: I’ve just tried Pytorch 2.0 with Cuda 11.8 and got the same error.</p>

Keith_Blaze: <p>The error message “CUDA driver error: invalid argument” indicates that there is an issue with the arguments being passed to a CUDA function. In this specific case, it could be caused by a few factors.</p>
<p>One possibility is that the size of the tensor being passed to the <code>prod()</code> function is too large to fit into the GPU memory. You can check the available memory on your GPU by running <code>torch.cuda.memory_allocated()</code> and <code>torch.cuda.max_memory_allocated()</code>. If the tensor is too large, you may need to reduce its size or consider using a larger GPU.</p>
<p>Another possibility is that the values in the tensor are not valid for the operation being performed. For example, if the tensor contains negative values and the operation requires positive values only, you may encounter this error. You can check the values in the tensor by printing it before calling the <code>prod()</code> function.</p>
<p>Additionally, this error could also be caused by a bug in the CUDA driver or an issue with the CUDA installation. In this case, you may need to reinstall the CUDA toolkit or update the driver to a newer version.</p>
<p>To better diagnose the issue, you can also try running the code on CPU instead of GPU by removing the <code>.cuda()</code> call and see if the error persists. Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<ol>
<li>Check the available memory on your GPU by running <code>torch.cuda.memory_allocated()</code> and <code>torch.cuda.max_memory_allocated()</code> to see if the tensor you are trying to process is too large for your GPU. If the tensor is too large, you may need to reduce its size or consider using a larger GPU.</li>
<li>Check the values in the tensor by printing it before calling the <code>prod()</code> function. Ensure that the tensor contains valid values for the operation being performed.</li>
<li>If the tensor is not too large and contains valid values, try updating your CUDA driver to the latest version or reinstalling the CUDA toolkit. You can find instructions for doing this on the NVIDIA website.</li>
<li>Alternatively, try running the code on CPU instead of GPU by removing the <code>.cuda()</code> call and see if the error persists. If the code runs without errors on CPU, it may be a problem with your GPU or CUDA installation.</li>
<li>If none of the above solutions work, try searching for the error message online to see if other people have encountered similar issues and found solutions. You could also post your code and error message on online forums such as Stack Overflow to get help from the community.</li>
</ol>

aliyesilkanat: <p>Thank you for your response.</p>
<p>Actually, I am able to train some neural networks on this configuration without any problem. So far, the only problem I see is the <code>prod</code> method on a Cuda tensor (CPU works fine).</p>
<p>I am using a cluster, and I have access to both V100 and A100 GPUs and can confirm that I receive this error on both. I also tried 3 different Cuda installations on 3 different conda environments, and they all have this problem. Lastly, I asked another user of the server to regenerate the error on the same setup (identical pytorch/cuda versions and gpu-also driver), but sadly it worked for this case. Therefore, I wonder what might prevent me from calling the <code>prod</code> method on my setup.</p>

AlphaBetaGamma96: <p>You could perhaps change from using <code>torch.Tensor</code> to something like <code>torch.as_tensor</code> as see if that works?</p>
<p>One other thing that you could try is to assign your data to a tensor, then on a new line move to cuda, then on a new line take the product.</p>
<p>For example change,</p>
<pre><code class="lang-auto">torch.as_tensor([[32, 32], [16, 16], [8, 8]]).cuda().prod(1)
</code></pre>
<p>to,</p>
<pre><code class="lang-auto">data = torch.as_tensor([[32, 32], [16, 16], [8, 8]]) 
data = data.cuda() #perhaps specify a device? 
result = data.prod(1)
</code></pre>
<p>You could specify a device ID, like <code>cuda(0)</code> or do <code>.to(device='cuda')</code> and see if that helps.</p>
<p>After having a look on StackOverflow, it seems the error occurs with the installed version of CUDA. If the conda envs keep giving the same error, you could try a virtual env with pip instead?</p>

aliyesilkanat: <p>Thanks for your response.</p>
<p>I tried your code suggestions, but none of them worked. On the cluster, I also have access to another conda, having a set of modules including pytorch-cuda combinations. Instead of using my conda environments, I tried pytorch-gpu/py3/1.13.0 with cuda/11.2 and pytorch-gpu/py3/2.0.0 with cuda/11.7.1 by <code>module load</code>, and the same problem still occurred.</p>
<p>I don’t understand what’s happening, but I won’t pursue it any more. Instead, I will convert the code as below and leave it like that.</p>
<pre><code class="lang-auto">import torch

a = torch.tensor([[32, 32], [16, 16], [8, 8]]).cuda()
result = torch.ones(a.shape[0]).cuda()

for i in range(a.shape[1]):
    result *= a[:, i]

print(result)
</code></pre>

javonh: <p>For anyone else facing this issue in the future, I had a similar problem but fixed it by adding .double() to my tensor, i.e.</p>
<pre><code class="lang-auto">torch.Tensor([[32, 32], [16, 16], [8, 8]]).double().cuda().prod(1)
</code></pre>
<p>You could also write it like this:</p>
<pre><code class="lang-auto">torch.Tensor([[32, 32], [16, 16], [8, 8]]).to("cuda", torch.double).prod(1)
</code></pre>
<p>Hopefully, this should fix your issue!</p>

aliyesilkanat: <p>Thank you for your response!</p>
<p>I can confirm it fixed the issue.</p>

ptrblck: <p>I don’t think any of these suggestions are a valid fix, but I’m also unable to reproduce the issue locally.<br>
In case you have any information how to reproduce the issue, it would be great it you could share these.</p>

RomanCast: <p>Hi, I also have the same issue, and I suspect we might be on the same cluster (using V100 at the moment). Calling <code>.double()</code> does not fix it for me however.</p>
<p>I have tried different versions of cuda (cuda/10.2 and cuda/11.2) but the result is the same. <code>.prod()</code> works on CPU but not on GPU. Every other operation works fine for me.</p>
<p>I noticed that I have this problem only with <code>torch==2.0.1</code> but <strong>not</strong> <code>torch==1.10.2</code> or <code>torch==1.8.1</code>.</p>
<p>Please tell me if you want me to give any additional info about the environment !</p>
<p><strong>EDIT</strong>: I can also confirm that I have the error with <code>torch==1.13.0</code> and <code>torch==1.12.1</code> but not <code>torch==1.11.0</code> (using cuda/11.2 in both cases)</p>

ptrblck: <p>I still cannot reproduce it with <code>2.0.1+cu117</code> and <code>2.0.1+cu118</code> on a V100 and get:</p>
<pre><code class="lang-python">tensor([1024.,  256.,   64.], device='cuda:0')
</code></pre>

aliyesilkanat: <p>Hi, Yes, I also think we are using the same cluster. I also have the same error on torch==1.13.0 but not on torch==1.11.0</p>

RomanCast: <p>I contacted the cluster admins and apparently this issue appears with my account but not other accounts, so I guess this is not really torch related. I will still write what was the issue here if I can find it.</p>
<p>Thanks for your help !</p>

ptrblck: <p>Yes, please let me know once you have more information about the issue as it’s really strange.</p>

RomanCast: <p>The cluster admins have found the solution, which was to clean <code>.cache/torch/kernels</code> which contained a bunch of files like the following:</p>
<p><code>reduction_prod_arch7.0_nvrtc11.2_sass_29385_edd9f40fe47533f6cc3bc6072826690d02ec24ef</code></p>
<p>It looks like cleaning it removed the problem altogether.</p>

ptrblck: <p>That’s really interesting and the first time I see that the cache cause issues!<br>
Thanks for sharing and I’ll keep it in mind in case similar issues pop up.</p>

Liahimz: <p>Hello, I have the same error on my NVIDIA RTX 6000, so my code is probably the same</p>
<pre><code class="lang-auto">a1 = torch.rand(1).cuda().prod(0)
</code></pre>
<p>Driver Version: 535.86.10    CUDA Version: 12.2 (all latest cuda toolkits and cudaNN are instaled)<br>
Everything installed without conda environments.<br>
And I got such message on output (definitely prod() is something that cause this error, without everything works fine)</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/mzingerenko/yolov5/test.py", line 5, in &lt;module&gt;
    a1 = torch.rand(1).cuda().prod(0)
RuntimeError: 
  #define POS_INFINITY __int_as_float(0x7f800000)
  #define INFINITY POS_INFINITY
  #define NEG_INFINITY __int_as_float(0xff800000)
  #define NAN __int_as_float(0x7fffffff)

  typedef long long int int64_t;
  typedef unsigned int uint32_t;
  typedef signed char int8_t;
  typedef unsigned char uint8_t;  // NOTE: this MUST be "unsigned char"! "char" is equivalent to "signed char"
  typedef short int16_t;
  static_assert(sizeof(int64_t) == 8, "expected size does not match");
  static_assert(sizeof(uint32_t) == 4, "expected size does not match");
  static_assert(sizeof(int8_t) == 1, "expected size does not match");
  constexpr int num_threads = 128;
  constexpr int thread_work_size = 4; // TODO: make template substitution once we decide where those vars live
  constexpr int block_work_size = thread_work_size * num_threads;
  //TODO use _assert_fail, because assert is disabled in non-debug builds
  #define ERROR_UNSUPPORTED_CAST assert(false);

  
  
  
  namespace std {
  
  using ::signbit;
  using ::isfinite;
  using ::isinf;
  using ::isnan;
  
  using ::abs;
  
  using ::acos;
  using ::acosf;
  using ::asin;
  using ::asinf;
  using ::atan;
  using ::atanf;
  using ::atan2;
  using ::atan2f;
  using ::ceil;
  using ::ceilf;
  using ::cos;
  using ::cosf;
  using ::cosh;
  using ::coshf;
  
  using ::exp;
  using ::expf;
  
  using ::fabs;
  using ::fabsf;
  using ::floor;
  using ::floorf;
  
  using ::fmod;
  using ::fmodf;
  
  using ::frexp;
  using ::frexpf;
  using ::ldexp;
  using ::ldexpf;
  
  using ::log;
  using ::logf;
  
  using ::log10;
  using ::log10f;
  using ::modf;
  using ::modff;
  
  using ::pow;
  using ::powf;
  
  using ::sin;
  using ::sinf;
  using ::sinh;
  using ::sinhf;
  
  using ::sqrt;
  using ::sqrtf;
  using ::tan;
  using ::tanf;
  
  using ::tanh;
  using ::tanhf;
  
  using ::acosh;
  using ::acoshf;
  using ::asinh;
  using ::asinhf;
  using ::atanh;
  using ::atanhf;
  using ::cbrt;
  using ::cbrtf;
  
  using ::copysign;
  using ::copysignf;
  
  using ::erf;
  using ::erff;
  using ::erfc;
  using ::erfcf;
  using ::exp2;
  using ::exp2f;
  using ::expm1;
  using ::expm1f;
  using ::fdim;
  using ::fdimf;
  using ::fmaf;
  using ::fma;
  using ::fmax;
  using ::fmaxf;
  using ::fmin;
  using ::fminf;
  using ::hypot;
  using ::hypotf;
  using ::ilogb;
  using ::ilogbf;
  using ::lgamma;
  using ::lgammaf;
  using ::llrint;
  using ::llrintf;
  using ::llround;
  using ::llroundf;
  using ::log1p;
  using ::log1pf;
  using ::log2;
  using ::log2f;
  using ::logb;
  using ::logbf;
  using ::lrint;
  using ::lrintf;
  using ::lround;
  using ::lroundf;
  
  using ::nan;
  using ::nanf;
  
  using ::nearbyint;
  using ::nearbyintf;
  using ::nextafter;
  using ::nextafterf;
  using ::remainder;
  using ::remainderf;
  using ::remquo;
  using ::remquof;
  using ::rint;
  using ::rintf;
  using ::round;
  using ::roundf;
  using ::scalbln;
  using ::scalblnf;
  using ::scalbn;
  using ::scalbnf;
  using ::tgamma;
  using ::tgammaf;
  using ::trunc;
  using ::truncf;
  
  } // namespace std
  
  

  // NB: Order matters for this macro; it is relied upon in
  // _promoteTypesLookup and the serialization format.
  // Note, some types have ctype as void because we don't support them in codegen
  #define AT_FORALL_SCALAR_TYPES_WITH_COMPLEX(_) \
  _(uint8_t, Byte) /* 0 */                               \
  _(int8_t, Char) /* 1 */                                \
  _(int16_t, Short) /* 2 */                              \
  _(int, Int) /* 3 */                                    \
  _(int64_t, Long) /* 4 */                               \
  _(at::Half, Half) /* 5 */                                  \
  _(float, Float) /* 6 */                                \
  _(double, Double) /* 7 */                              \
  _(std::complex&lt;at::Half&gt;, ComplexHalf) /* 8 */        \
  _(std::complex&lt;float&gt;, ComplexFloat) /* 9 */                          \
  _(std::complex&lt;double&gt;, ComplexDouble) /* 10 */                         \
  _(bool, Bool) /* 11 */                                 \
  _(void, QInt8) /* 12 */                          \
  _(void, QUInt8) /* 13 */                        \
  _(void, QInt32) /* 14 */                        \
  _(at::BFloat16, BFloat16) /* 15 */                             \

  #define AT_FORALL_SCALAR_TYPES_WITH_COMPLEX_EXCEPT_QINT(_)       \
  _(uint8_t, Byte)                                                 \
  _(int8_t, Char)                                                  \
  _(int16_t, Short)                                                \
  _(int, Int)                                                      \
  _(int64_t, Long)                                                 \
  _(at::Half, Half)                                                \
  _(float, Float)                                                  \
  _(double, Double)                                                \
  _(std::complex&lt;at::Half&gt;, ComplexHalf)                           \
  _(std::complex&lt;float&gt;, ComplexFloat)                             \
  _(std::complex&lt;double&gt;, ComplexDouble)                           \
  _(bool, Bool)                                                    \
  _(at::BFloat16, BFloat16)


  enum class ScalarType : int8_t {
  #define DEFINE_ENUM(_1, n) n,
  AT_FORALL_SCALAR_TYPES_WITH_COMPLEX(DEFINE_ENUM)
  #undef DEFINE_ENUM
      Undefined,
  NumOptions
  };

  template &lt;typename T, int size&gt;
  struct Array {
  T data[size];

  __device__ T operator[](int i) const {
      return data[i];
  }
  __device__ T&amp; operator[](int i) {
      return data[i];
  }
  Array() = default;
  Array(const Array&amp;) = default;
  Array&amp; operator=(const Array&amp;) = default;
  __device__ Array(T x) {
    for (int i = 0; i &lt; size; i++) {
      data[i] = x;
    }
  }
  };

  
  
  
  
  



  template &lt;typename T&gt;
  struct DivMod {
  T div;
  T mod;

  __device__ DivMod(T _div, T _mod) {
      div = _div;
      mod = _mod;
  }
  };

  //&lt;unsigned int&gt;
  struct IntDivider {
  IntDivider() = default;

  __device__ inline unsigned int div(unsigned int n) const {
  unsigned int t = __umulhi(n, m1);
  return (t + n) &gt;&gt; shift;
  }

  __device__ inline unsigned int mod(unsigned int n) const {
  return n - div(n) * divisor;
  }

  __device__ inline DivMod&lt;unsigned int&gt; divmod(unsigned int n) const {
  unsigned int q = div(n);
  return DivMod&lt;unsigned int&gt;(q, n - q * divisor);
  }

  unsigned int divisor;  // d above.
  unsigned int m1;  // Magic number: m' above.
  unsigned int shift;  // Shift amounts.
  };

  template &lt;int NARGS&gt;
  struct TrivialOffsetCalculator {
    // The offset for each argument. Wrapper around fixed-size array.
    // The offsets are in # of elements, not in bytes.
    Array&lt;unsigned int, NARGS&gt; get(unsigned int linear_idx) const {
      Array&lt;unsigned int, NARGS&gt; offsets;
      #pragma unroll
      for (int arg = 0; arg &lt; NARGS; arg++) {
        offsets[arg] = linear_idx;
      }
      return offsets;
    }
  };

  template&lt;int NARGS&gt;
  struct OffsetCalculator {
  OffsetCalculator() = default;
  __device__ __forceinline__ Array&lt;unsigned int, NARGS&gt; get(unsigned int linear_idx) const {
      Array&lt;unsigned int, NARGS&gt; offsets;
      #pragma unroll
      for (int arg = 0; arg &lt; NARGS; ++arg) {
      offsets[arg] = 0;
      }

      #pragma unroll
      for (int dim = 0; dim &lt; 25; ++dim) {
      if (dim == dims) {
          break;
      }

      auto divmod = sizes_[dim].divmod(linear_idx);
      linear_idx = divmod.div;

      #pragma unroll
      for (int arg = 0; arg &lt; NARGS; ++arg) {
          offsets[arg] += divmod.mod * strides_[dim][arg];
      }
      //printf("offset calc thread dim size stride offset %d %d %d %d %d %d %d %d\n",
      //threadIdx.x, dim, sizes_[dim].divisor, strides_[dim][0], offsets[0], linear_idx, divmod.div, divmod.mod);
      }
      return offsets;
  }

    int dims;
    IntDivider sizes_[25];
    // NOTE: this approach will not support nInputs == 0
    unsigned int strides_[25][NARGS];
  };



  #define C10_HOST_DEVICE __host__ __device__
  #define C10_DEVICE __device__

  template &lt;typename T&gt;
  __device__ __forceinline__ T WARP_SHFL_DOWN(T value, unsigned int delta, int width = warpSize, unsigned int mask = 0xffffffff)
  {
    return __shfl_down_sync(mask, value, delta, width);
  }


  #if 0
  template &lt;typename T&gt;
  __device__ __forceinline__ std::complex&lt;T&gt; WARP_SHFL_DOWN(std::complex&lt;T&gt; value, unsigned int delta, int width = warpSize, unsigned int mask = 0xffffffff)
  {
    return std::complex&lt;T&gt;(
        __shfl_down_sync(mask, value.real(), delta, width),
        __shfl_down_sync(mask, value.imag(), delta, width));
  }
  #endif

  // aligned vector generates vectorized load/store on CUDA
  template&lt;typename scalar_t, int vec_size&gt;
  struct alignas(sizeof(scalar_t) * vec_size) aligned_vector {
    scalar_t val[vec_size];
  };


  C10_HOST_DEVICE static void reduce_fraction(size_t &amp;numerator, size_t &amp;denominator) {
    // get GCD of num and denom using Euclid's algorithm.
    // Can replace this with std::gcd if we ever support c++17.
    size_t a = denominator;
    size_t b = numerator;
    while (b != 0) {
        a %= b;
        // swap(a,b)
        size_t tmp = a;
        a = b;
        b = tmp;
    }

    // a is now the GCD
    numerator /= a;
    denominator /= a;
  }




  struct ReduceConfig {
  //has to match host-side ReduceConfig in the eager code
  static constexpr int BLOCK_X = 0;
  static constexpr int BLOCK_Y = 1;
  static constexpr int CTA = 2;

  static constexpr int input_vec_size = 4;
  int element_size_bytes;
  int num_inputs;
  int num_outputs;
  int step_input = 1;
  int step_output = 1;
  int ctas_per_output = 1;
  int input_mult[3] = {0, 0, 0};
  int output_mult[2] = {0, 0};

  int block_width;
  int block_height;
  int num_threads;

  bool vectorize_input = false;
  int output_vec_size = 1;

  C10_HOST_DEVICE bool should_block_x_reduce() const {
    return input_mult[BLOCK_X] != 0;
  }

  C10_HOST_DEVICE bool should_block_y_reduce() const {
    return input_mult[BLOCK_Y] != 0;
  }

  C10_HOST_DEVICE bool should_global_reduce() const {
    return input_mult[CTA] != 0;
  }

  C10_DEVICE bool should_store(int output_idx) const {
    return output_idx &lt; num_outputs &amp;&amp;
      (!should_block_x_reduce() || threadIdx.x == 0) &amp;&amp;
      (!should_block_y_reduce() || threadIdx.y == 0);
  }

  C10_DEVICE bool should_reduce_tail() const {
    return (!should_block_y_reduce() || threadIdx.y == 0) &amp;&amp;
      (!should_global_reduce() || blockIdx.y == 0);
  }

  C10_HOST_DEVICE int input_idx() const {
    int lane = threadIdx.x;
    int warp = threadIdx.y;
    int cta2 = blockIdx.y;
    return (lane * input_mult[BLOCK_X] +
            warp * input_mult[BLOCK_Y] +
            cta2 * input_mult[CTA]);
  }

  template &lt;int output_vec_size&gt;
  C10_HOST_DEVICE int output_idx() const {
    int lane = threadIdx.x;
    int warp = threadIdx.y;
    int cta1 = blockIdx.x;
    return (lane * output_mult[BLOCK_X] +
            warp * output_mult[BLOCK_Y] +
            cta1 * step_output) * output_vec_size;
  }

  C10_DEVICE int shared_memory_offset(int offset) const {
    return threadIdx.x + (threadIdx.y + offset) * blockDim.x;
  }

  C10_DEVICE int staging_memory_offset(int cta2) const {
    int offset = cta2 + blockIdx.x * gridDim.y;
    if (!should_block_x_reduce()) {
      offset = threadIdx.x + offset * blockDim.x;
    }
    return offset;
  }


  };


//TODO this will need to be different for more generic reduction functions
namespace reducer {

  using scalar_t = float;
  using arg_t = float;
  using out_scalar_t = float;


  inline __device__ arg_t combine(arg_t a, arg_t b) { return a * b; }

  inline __device__ out_scalar_t project(arg_t arg) {
    return (out_scalar_t) arg;
  }

  inline __device__ arg_t warp_shfl_down(arg_t arg, int offset) {
    return WARP_SHFL_DOWN(arg, offset);
  }

  inline __device__ arg_t translate_idx(arg_t acc, int64_t /*idx*/) {
    return acc;
  }

  // wrap a normal reduction that ignores the index
  inline __device__ arg_t reduce(arg_t acc, arg_t val, int64_t idx) {
     return combine(acc, val);
  }
}


struct ReduceJitOp {
  using scalar_t = float;
  using arg_t = float;
  using out_scalar_t = float;

  using InputCalculator = OffsetCalculator&lt;1&gt;;
  using OutputCalculator = OffsetCalculator&lt;2&gt;;

//   static constexpr bool can_accumulate_in_output =
//     std::is_convertible&lt;arg_t, out_scalar_t&gt;::value
//     &amp;&amp; std::is_convertible&lt;out_scalar_t, arg_t&gt;::value;

  static constexpr int input_vec_size = ReduceConfig::input_vec_size;

  arg_t ident;
  ReduceConfig config;
  InputCalculator input_calc;
  OutputCalculator output_calc;
  const void* src;
  const char* dst[2]; //it accepts at most two destinations
  // acc_buf used for accumulation among sub Tensor Iterator when accumulation on
  // output is not permissible
  void* acc_buf;
  // cta_buf used for accumulation between blocks during global reduction
  void* cta_buf;
  int* semaphores;
  int64_t base_idx;
  bool accumulate;
  bool final_output;
  int noutputs;


  C10_DEVICE void run() const {
    extern __shared__ char shared_memory[];
    uint32_t output_idx = config.output_idx&lt;1&gt;();
    uint32_t input_idx = config.input_idx();
    auto base_offsets1 = output_calc.get(output_idx)[1];

    using arg_vec_t = Array&lt;arg_t, 1&gt;;
    arg_vec_t value;

    if (output_idx &lt; config.num_outputs &amp;&amp; input_idx &lt; config.num_inputs) {
      const scalar_t* input_slice = (const scalar_t*)((const char*)src + base_offsets1);

      value = thread_reduce&lt;1&gt;(input_slice);
    }

    if (config.should_block_y_reduce()) {
      value = block_y_reduce&lt;1&gt;(value, shared_memory);
    }
    if (config.should_block_x_reduce()) {
      value = block_x_reduce&lt;1&gt;(value, shared_memory);
    }

    using out_ptr_vec_t = Array&lt;out_scalar_t*, 1&gt;;
    using offset_vec_t = Array&lt;uint32_t, 1&gt;;
    offset_vec_t base_offsets;
    out_ptr_vec_t out;

    #pragma unroll
    for (int i = 0; i &lt; 1; i++) {
      base_offsets[i] = output_calc.get(output_idx + i)[0];
      out[i] = (out_scalar_t*)((char*)dst[0] + base_offsets[i]);
    }

    arg_vec_t* acc = nullptr;
    if (acc_buf != nullptr) {
      size_t numerator = sizeof(arg_t);
      size_t denominator = sizeof(out_scalar_t);
      reduce_fraction(numerator, denominator);
      acc = (arg_vec_t*)((char*)acc_buf + (base_offsets[0] * numerator / denominator));
    }

    if (config.should_global_reduce()) {
      value = global_reduce&lt;1&gt;(value, acc, shared_memory);
    } else if (config.should_store(output_idx)) {
      if (accumulate) {
        #pragma unroll
        for (int i = 0; i &lt; 1; i++) {
          value[i] = reducer::translate_idx(value[i], base_idx);
        }
      }

      if (acc == nullptr) {
        if (accumulate) {
          value = accumulate_in_output&lt;1&gt;(out, value);
        }
        if (final_output) {
          set_results_to_output&lt;1&gt;(value, base_offsets);
        } else {
          #pragma unroll
          for (int i = 0; i &lt; 1; i++) {
            *(out[i]) = get_accumulated_output(out[i], value[i]);
          }
        }
      } else {
        if (accumulate) {
          #pragma unroll
          for (int i = 0; i &lt; 1; i++) {
            value[i] = reducer::combine((*acc)[i], value[i]);
          }
        }
        if (final_output) {
          set_results_to_output&lt;1&gt;(value, base_offsets);
        } else {
          *acc = value;
        }
      }
    }
  }

  template &lt;int output_vec_size&gt;
  C10_DEVICE Array&lt;arg_t, output_vec_size&gt; thread_reduce(const scalar_t* data) const {
    if (config.vectorize_input) {
      assert(output_vec_size == 1);
      // reduce at the header of input_slice where memory is not aligned,
      // so that thread_reduce will have an aligned memory to work on.
      return {input_vectorized_thread_reduce_impl(data)};
    } else {
      uint32_t element_stride = input_calc.strides_[0][0] / sizeof(scalar_t);
      bool is_contiguous = (input_calc.dims == 1 &amp;&amp; element_stride == 1);
      if (is_contiguous) {
        return thread_reduce_impl&lt;output_vec_size&gt;(data, [](uint32_t idx) { return idx; });
      } else if (input_calc.dims == 1) {
        return thread_reduce_impl&lt;output_vec_size&gt;(data, [&amp;](uint32_t idx) { return idx * element_stride; });
      } else {
        return thread_reduce_impl&lt;output_vec_size&gt;(data, [&amp;](uint32_t idx) { return input_calc.get(idx)[0] / sizeof(scalar_t); });
      }
    }
  }

  C10_DEVICE arg_t input_vectorized_thread_reduce_impl(const scalar_t* data) const {
    uint32_t end = config.num_inputs;

    // Handle the head of input slice where data is not aligned
    arg_t value = ident;
    constexpr int align_bytes = alignof(aligned_vector&lt;scalar_t, input_vec_size&gt;);
    constexpr int align_elements = align_bytes / sizeof(scalar_t);
    int shift = ((int64_t)data) % align_bytes / sizeof(scalar_t);
    if (shift &gt; 0) {
      data -= shift;
      end += shift;
      if(threadIdx.x &gt;= shift &amp;&amp; threadIdx.x &lt; align_elements &amp;&amp; config.should_reduce_tail()){
        value = reducer::reduce(value, data[threadIdx.x], threadIdx.x - shift);
      }
      end -= align_elements;
      data += align_elements;
      shift = align_elements - shift;
    }

    // Do the vectorized reduction
    using load_t = aligned_vector&lt;scalar_t, input_vec_size&gt;;

    uint32_t idx = config.input_idx();
    const uint32_t stride = config.step_input;

    // Multiple accumulators to remove dependency between unrolled loops.
    arg_t value_list[input_vec_size];
    value_list[0] = value;

    #pragma unroll
    for (int i = 1; i &lt; input_vec_size; i++) {
      value_list[i] = ident;
    }

    scalar_t values[input_vec_size];

    load_t *values_vector = reinterpret_cast&lt;load_t*&gt;(&amp;values[0]);

    while (idx * input_vec_size + input_vec_size - 1 &lt; end) {
      *values_vector = reinterpret_cast&lt;const load_t*&gt;(data)[idx];
      #pragma unroll
      for (uint32_t i = 0; i &lt; input_vec_size; i++) {
        value_list[i] = reducer::reduce(value_list[i], values[i], shift + idx * input_vec_size + i);
      }
      idx += stride;
    }

    // tail
    uint32_t tail_start = end - end % input_vec_size;
    if (config.should_reduce_tail()) {
      int idx = tail_start + threadIdx.x;
      if (idx &lt; end) {
        value_list[0] = reducer::reduce(value_list[0], data[idx], idx + shift);
      }
    }

    // combine accumulators
    #pragma unroll
    for (int i = 1; i &lt; input_vec_size; i++) {
      value_list[0] = reducer::combine(value_list[0], value_list[i]);
    }
    return value_list[0];
  }

  template &lt;int output_vec_size, typename offset_calc_t&gt;
  C10_DEVICE Array&lt;arg_t, output_vec_size&gt; thread_reduce_impl(const scalar_t* data_, offset_calc_t calc) const {
    uint32_t idx = config.input_idx();
    const uint32_t end = config.num_inputs;
    const uint32_t stride = config.step_input;
    const int vt0=4;

    using arg_vec_t = Array&lt;arg_t, output_vec_size&gt;;
    using load_t = aligned_vector&lt;scalar_t, output_vec_size&gt;;
    const load_t* data = reinterpret_cast&lt;const load_t*&gt;(data_);

    // Multiple accumulators to remove dependency between unrolled loops.
    arg_vec_t value_list[vt0];

    #pragma unroll
    for (int i = 0; i &lt; vt0; i++) {
      #pragma unroll
      for (int j = 0; j &lt; output_vec_size; j++) {
        value_list[i][j] = ident;
      }
    }

    load_t values[vt0];

    while (idx + (vt0 - 1) * stride &lt; end) {
      #pragma unroll
      for (uint32_t i = 0; i &lt; vt0; i++) {
        values[i] = data[calc(idx + i * stride) / output_vec_size];
      }
      #pragma unroll
      for (uint32_t i = 0; i &lt; vt0; i++) {
        #pragma unroll
        for (uint32_t j = 0; j &lt; output_vec_size; j++) {
          value_list[i][j] = reducer::reduce(value_list[i][j], values[i].val[j], idx + i * stride);
        }
      }
      idx += stride * vt0;
    }

    // tail
    int idx_ = idx;
    #pragma unroll
    for (uint32_t i = 0; i &lt; vt0; i++) {
      if (idx &gt;= end) {
        break;
      }
      values[i] = data[calc(idx) / output_vec_size];
      idx += stride;
    }
    idx = idx_;
    #pragma unroll
    for (uint32_t i = 0; i &lt; vt0; i++) {
      if (idx &gt;= end) {
        break;
      }
      #pragma unroll
      for (uint32_t j = 0; j &lt; output_vec_size; j++) {
        value_list[i][j] = reducer::reduce(value_list[i][j], values[i].val[j], idx);
      }
      idx += stride;
    }

    // combine accumulators
    #pragma unroll
    for (int i = 1; i &lt; vt0; i++) {
      #pragma unroll
      for (uint32_t j = 0; j &lt; output_vec_size; j++) {
        value_list[0][j] = reducer::combine(value_list[0][j], value_list[i][j]);
      }
    }
    return value_list[0];
  }
  template &lt;int output_vec_size&gt;
  C10_DEVICE Array&lt;arg_t, output_vec_size&gt; block_x_reduce(Array&lt;arg_t, output_vec_size&gt; value, char* shared_memory) const {
    using args_vec_t = Array&lt;arg_t, output_vec_size&gt;;
    int dim_x = blockDim.x;
    args_vec_t* shared = (args_vec_t*)shared_memory;
    if (dim_x &gt; warpSize) {
      int address_base = threadIdx.x + threadIdx.y*blockDim.x;
      shared[address_base] = value;
      for (int offset = dim_x/2; offset &gt;= warpSize; offset &gt;&gt;= 1) {
        __syncthreads();
        if (threadIdx.x &lt; offset &amp;&amp; threadIdx.x + offset &lt; blockDim.x) {
          args_vec_t other = shared[address_base + offset];
          #pragma unroll
          for (int i = 0; i &lt; output_vec_size; i++) {
            value[i] = reducer::combine(value[i], other[i]);
          }
          shared[address_base] = value;
        }
      }
      dim_x = warpSize;
    }

    __syncthreads();

    for (int offset = 1; offset &lt; dim_x; offset &lt;&lt;= 1) {
      #pragma unroll
      for (int i = 0; i &lt; output_vec_size; i++) {
        arg_t other = reducer::warp_shfl_down(value[i], offset);
        value[i] = reducer::combine(value[i], other);
      }
    }
    return value;
  }

  template &lt;int output_vec_size&gt;
  C10_DEVICE Array&lt;arg_t, output_vec_size&gt; block_y_reduce(Array&lt;arg_t, output_vec_size&gt; value, char* shared_memory) const {
    using args_vec_t = Array&lt;arg_t, output_vec_size&gt;;
    args_vec_t* shared = (args_vec_t*)shared_memory;
    shared[config.shared_memory_offset(0)] = value;
    for (int offset = blockDim.y / 2; offset &gt; 0; offset &gt;&gt;= 1) {
      __syncthreads();
      if (threadIdx.y &lt; offset &amp;&amp; threadIdx.y + offset &lt; blockDim.y) {
        args_vec_t other = shared[config.shared_memory_offset(offset)];
        #pragma unroll
        for (int i = 0; i &lt; output_vec_size; i++) {
          value[i] = reducer::combine(value[i], other[i]);
        }
        shared[config.shared_memory_offset(0)] = value;
      }
    }
    return value;
  }
  

  C10_DEVICE bool mark_block_finished() const {
    __shared__ bool is_last_block_done_shared;

    __syncthreads();
    if (threadIdx.x == 0 &amp;&amp; threadIdx.y == 0) {
      int prev_blocks_finished = atomicAdd(&amp;semaphores[blockIdx.x], 1);
      is_last_block_done_shared = (prev_blocks_finished == gridDim.y - 1);
    }

    __syncthreads();

    return is_last_block_done_shared;
  }

  template &lt;int output_vec_size&gt;
  C10_DEVICE Array&lt;arg_t, output_vec_size&gt; accumulate_in_output(
    Array&lt;out_scalar_t*, output_vec_size&gt; out,
    Array&lt;arg_t, output_vec_size&gt; value
  ) const {
    Array&lt;arg_t, output_vec_size&gt; ret;
    #pragma unroll
    for (int i = 0; i &lt; output_vec_size; i++) {
      ret[i] = reducer::combine(*(out[i]), value[i]);
    }
    return ret;
  }


  C10_DEVICE out_scalar_t get_accumulated_output(
    out_scalar_t* out, arg_t value
  ) const {
    assert(!final_output);
    return (out_scalar_t)value;
  }

  template&lt;class T&gt;
  C10_DEVICE void set_results(const T x, const uint32_t base_offset) const {
    assert(noutputs == 1);
    auto res = (out_scalar_t*)((char*)dst[0] + base_offset);
    *res = x;
  }

//TODO - multi-output reduction - we won't be able to use thrust::pair
//just explicitly specify typed output reads/writes
//Currently implemented for max of two outputs
//   template&lt;class T1, class T2&gt;
//   C10_DEVICE void set_results(const thrust::pair&lt;T1, T2&gt; x, const index_t base_offset) const {
//     if (noutputs &gt;= 1) {
//       auto res0 = (T1*)((char*)dst[0] + base_offset);
//       *res0 = x.first;
//     }
//     if (noutputs &gt;= 2) {
//       // base offset is computed assuming element size being sizeof(T1), so we need to make a
//       // correction to obtain the correct base offset
//       auto res1 = (T2*) ((char *) dst[1] + base_offset / sizeof(T1) * sizeof(T2));
//       *res1 = x.second;
//     }
//   }

  template &lt;int output_vec_size&gt;
  C10_DEVICE void set_results_to_output(Array&lt;arg_t, output_vec_size&gt; value, Array&lt;uint32_t, output_vec_size&gt; base_offset) const {
    assert(final_output);
    #pragma unroll
    for (int i = 0; i &lt; output_vec_size; i++) {
      set_results(reducer::project(value[i]), base_offset[i]);
    }
  }

  template &lt;int output_vec_size&gt;
  C10_DEVICE Array&lt;arg_t, output_vec_size&gt; global_reduce(Array&lt;arg_t, output_vec_size&gt; value, Array&lt;arg_t, output_vec_size&gt; *acc, char* shared_memory) const {
    using arg_vec_t = Array&lt;arg_t, output_vec_size&gt;;
    using out_ptr_vec_t = Array&lt;out_scalar_t*, output_vec_size&gt;;
    using offset_vec_t = Array&lt;uint32_t, output_vec_size&gt;;

    arg_vec_t* reduce_buffer = (arg_vec_t*)cta_buf;
    uint32_t output_idx = config.output_idx&lt;output_vec_size&gt;();
    offset_vec_t base_offsets;
    out_ptr_vec_t out;

    #pragma unroll
    for (int i = 0; i &lt; output_vec_size; i++) {
      base_offsets[i] = output_calc.get(output_idx + i)[0];
      out[i] = (out_scalar_t*)((char*)dst[0] + base_offsets[i]);
    }

    bool should_store = config.should_store(output_idx);
    if (should_store) {
      uint32_t offset = config.staging_memory_offset(blockIdx.y);
      reduce_buffer[offset] = value;
    }

    __threadfence(); // make sure writes are globally visible
    __syncthreads(); // if multiple warps in this block wrote to staging, make sure they're all done
    bool is_last_block_done = mark_block_finished();

    if (is_last_block_done) {
      value = ident;
      if (config.should_block_x_reduce()) {
        uint32_t input_offset = threadIdx.x + threadIdx.y * blockDim.x;
        uint32_t step = blockDim.x * blockDim.y;
        for (; input_offset &lt; config.ctas_per_output; input_offset += step) {
          uint32_t idx = config.staging_memory_offset(input_offset);
          arg_vec_t next = reduce_buffer[idx];
          #pragma unroll
          for (int i = 0; i &lt; output_vec_size; i++) {
            value[i] = reducer::combine(value[i], next[i]);
          }
        }
      } else {
        uint32_t input_offset = threadIdx.y;
        uint32_t step = blockDim.y;
        for (; input_offset &lt; config.ctas_per_output; input_offset += step) {
          uint32_t idx = config.staging_memory_offset(input_offset);
          arg_vec_t next = reduce_buffer[idx];
          #pragma unroll
          for (int i = 0; i &lt; output_vec_size; i++) {
            value[i] = reducer::combine(value[i], next[i]);
          }
        }
      }
      value = block_y_reduce(value, shared_memory);
      if (config.should_block_x_reduce()) {
        value = block_x_reduce&lt;output_vec_size&gt;(value, shared_memory);
      }
      if (should_store) {
        if (accumulate) {
          #pragma unroll
          for (int i = 0; i &lt; output_vec_size; i++) {
            value[i] = reducer::translate_idx(value[i], base_idx);
          }
        }

        if (acc == nullptr) {
          if (accumulate) {
            value = accumulate_in_output&lt;output_vec_size&gt;(out, value);
          }
          if (final_output) {
            set_results_to_output&lt;output_vec_size&gt;(value, base_offsets);
          } else {
            #pragma unroll
            for (int i = 0; i &lt; output_vec_size; i++) {
              *(out[i]) = get_accumulated_output(out[i], value[i]);
            }
          }
        } else {
          if (accumulate) {
            #pragma unroll
            for (int i = 0; i &lt; output_vec_size; i++) {
              value[i] = reducer::combine((*acc)[i], value[i]);
            }
          }
          if (final_output) {
            set_results_to_output&lt;output_vec_size&gt;(value, base_offsets);
          } else {
            *acc = value;
          }
        }
      }
    }

    return value;
  }
};

extern "C"
__launch_bounds__(512, 4)
__global__ void reduction_prod_kernel(ReduceJitOp r){
  r.run();
}
nvrtc: error: invalid value for --gpu-architecture (-arch)
</code></pre>
<p>And I don’t have any files inside .cache/torch/kernels so nothing to clean.</p>