Relation between at::Half and __half

sakura: <p>Hi, we want to create <strong>atomicMax</strong> function with __half inputs by following <a href="https://github.com/rusty1s/pytorch_scatter/blob/master/csrc/cuda/atomics.cuh" rel="noopener nofollow ugc">pytorch_scatter/atomics.cuh at master</a>. The detailed codes are</p>
<pre><code>inline __device__ void operator()(at::Half *address, at::Half val) {       \
  unsigned int *address_as_ui =                                            \
      (unsigned int *)((char *)address - ((size_t)address &amp; 2));           \
  unsigned int old = *address_as_ui;                                       \
  unsigned int assumed;                                                    \
                                                                           \
  do {                                                                     \
    assumed = old;                                                         \
    at::Half hsum;                                                         \
    hsum.x = (size_t)address &amp; 2 ? (old &gt;&gt; 16) : (old &amp; 0xffff);           \
    hsum = OP(hsum, val);                                                  \
    old = (size_t)address &amp; 2 ? (old &amp; 0xffff) | (hsum.x &lt;&lt; 16)            \
                              : (old &amp; 0xffff0000) | hsum.x;               \
    old = atomicCAS(address_as_ui, assumed, old);                          \
  } while (assumed != old);                                                \
}                                                                          \
</code></pre>
<p>When it comes to the <code>__half</code> type, however, we do not how to obtain the <code>hsum.x</code>. Are there some ways to convert <code>as::Half</code> and <code>__half</code> each other?</p>
<p>Also, how could we implement <code>OP</code> as max to deal with <code>__half</code> inputs?</p>
<p>Thanks : )</p>

eqy: <p>It’s a bit clunky, but I believe simply:<br>
<code>*reinterpret_cast&lt;__half*&gt;(&amp;myathalf);</code><br>
and<br>
<code>*reinterpret_cast&lt;at::Half*&gt;(&amp;myhalf);</code><br>
would work. To convert pointers you would simply drop the dereference.</p>

sakura: <p>Thanks for your advice. Actually, we want to create atomicMax just in normal <code>.cu</code> file without <code>at::Half</code> defination. So, are there some ways to correctly replace  <code>at::Half</code> by <code>__half</code> in above function?</p>
<p>I am not familiar with <a href="https://github.com/pytorch/pytorch/blob/master/c10/util/Half.h#L376" rel="noopener nofollow ugc">pytorch/Half.h at master</a> but simple replacement would bring error <code>class "__half" has no member "x"</code></p>

eqy: <p>What happens if you just use the value directly (remove the <code>.x</code>), if hsum is type <code>__half</code>?</p>

sakura: <p>Here are my modified codes</p>
<pre><code class="lang-auto">  unsigned int *address_as_ui = (unsigned int *)((char *)address - ((size_t)address &amp; 2));
  unsigned int old = *address_as_ui;
  unsigned int assumed;
  do {
    assumed = old;
    __half hsum;
    hsum = reinterpret_cast&lt;size_t&gt;(address) &amp; 2 ? (old &gt;&gt; 16) : (old &amp; 0xffff);
    hsum = __float2half(fmaxf(__half2float(hsum), __half2float(val)));
    old = (size_t)address &amp; 2 ? (old &amp; 0xffff) | (hsum &lt;&lt; 16)
                              : (old &amp; 0xffff0000) | hsum;
    old = atomicCAS(address_as_ui, assumed, old);
  } while (assumed != old);
</code></pre>
<p>The error appears in two <code>hsum</code> in <code>old</code> calculation:</p>
<pre><code class="lang-auto">test.cu(57): error: more than one conversion function from "__half" to a built-in type applies:
            function "__half::operator short() const"
/cuda-11.4/include/cuda_fp16.hpp(222): here
            function "__half::operator unsigned short() const"
/cuda-11.4/include/cuda_fp16.hpp(225): here
            function "__half::operator int() const"
/cuda-11.4/include/cuda_fp16.hpp(228): here
            function "__half::operator unsigned int() const"
/cuda-11.4/include/cuda_fp16.hpp(231): here
            function "__half::operator long long() const"
/cuda-11.4/include/cuda_fp16.hpp(234): here
            function "__half::operator unsigned long long() const"
/cuda-11.4/include/cuda_fp16.hpp(237): here
            function "__half::operator __nv_bool() const"
/cuda-11.4/include/cuda_fp16.hpp(241): here
</code></pre>

eqy: <p>What happens if you explicitly cast hsum (and/or) old to <code>short</code> as <code>x</code> would be in the original implementation?</p>

sakura: <p>Do you mean</p>
<pre><code class="lang-auto">  unsigned int *address_as_ui = (unsigned int *)((char *)address - ((size_t)address &amp; 2));
  unsigned int old = *address_as_ui;
  unsigned int assumed;
  do {
    assumed = old;
    __half hsum;
    hsum = reinterpret_cast&lt;size_t&gt;(address) &amp; 2 ? (old &gt;&gt; 16) : (old &amp; 0xffff);
    hsum = __float2half(fmaxf(__half2float(hsum), __half2float(val)));
    old = (size_t)address &amp; 2 ? (old &amp; 0xffff) | (__half_as_ushort(hsum) &lt;&lt; 16)
                              : (old &amp; 0xffff0000) | __half_as_ushort(hsum);
    old = atomicCAS(address_as_ui, assumed, old);
  } while (assumed != old);
</code></pre>
<p>This works and the compile error disappears. But the calculated result seems wrong…</p>

eqy: <p>Would it make a difference if it was converted to a signed short?</p>

sakura: <p>I use <code>__half_as_short</code> to replace <code>__half_as_ushort</code> but the calculation is still wrong.<br>
Now we have</p>
<pre><code class="lang-auto">__device__ static void atomicMax(__half* address, __half val){

  unsigned int *address_as_ui = (unsigned int *)((char *)address - ((size_t)address &amp; 2));
  unsigned int old = *address_as_ui;
  unsigned int assumed;
  do {
    assumed = old;
    __half hsum;
    hsum = reinterpret_cast&lt;size_t&gt;(address) &amp; 2 ? (old &gt;&gt; 16) : (old &amp; 0xffff);
    hsum = __float2half(fmaxf(__half2float(hsum), __half2float(val)));
    old = (size_t)address &amp; 2 ? (old &amp; 0xffff) | (__half_as_short(hsum) &lt;&lt; 16)
                              : (old &amp; 0xffff0000) | __half_as_short(hsum);
    old = atomicCAS(address_as_ui, assumed, old);
  } while (assumed != old);
}
</code></pre>

sakura: <p>At the same time,  I found <a href="https://github.com/NVIDIA/cutlass/issues/4" rel="noopener nofollow ugc">error: more than one conversion function from “half” to a built-in type applies · Issue #4 · NVIDIA/cutlass (github.com)</a>. I try to add</p>
<pre><code class="lang-auto">set(CMAKE_CUDA_ARCHITECTURES 86)
or
set(CUDA_ARCHITECTURE_FLAGS "86")
</code></pre>
<p>in my CMakeLists.txt but still does not work. What is the correct way to use above command in cmake?</p>