[CUDA/MSVC][Suggestion] ROI Pool half-precision build error due to ambiguous comparison

kimchioverfit: <p>Hi all,</p>
<p>I ran into a build issue when compiling the ROI Pool CUDA kernel on <strong>Windows + MSVC + CUDA</strong> with <code>T=half</code>. The kernel fails to compile due to an <strong>ambiguous comparison operator</strong>.</p>
<h3><a name="p-476255-environment-1" class="anchor" href="#p-476255-environment-1"></a>Environment</h3>
<ul>
<li>
<p>OS: Windows 10/11</p>
</li>
<li>
<p>Compiler: MSVC 19.x (Visual Studio 2022)</p>
</li>
<li>
<p>CUDA: 12.x</p>
</li>
<li>
<p>PyTorch / torchvision: Libtorch(2.3.1) + torchvision csrc</p>
</li>
</ul>
<h3><a name="p-476255-problematic-code-2" class="anchor" href="#p-476255-problematic-code-2"></a>Problematic code</h3>
<p>In <code>torchvision/csrc/ops/cuda/roi_pool_kernel.cu</code>, inside <code>roi_pool_forward_kernel_impl</code>:</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">if (offset_input[input_index] &gt;maxval) {
          maxval = offset_input[input_index];
          maxidx = input_index;
        }
</code></pre>
<h3><a name="p-476255-likely-cause-3" class="anchor" href="#p-476255-likely-cause-3"></a>Likely cause</h3>
<ul>
<li>For <code>__half</code> / <code>c10::Half</code>, the availability of operator overloads depends on macros such as <code>__CUDA_NO_HALF_OPERATORS__</code> and <code>__CUDA_NO_HALF_CONVERSIONS__</code>.</li>
<li>On MSVC+NVCC, the implicit promotion from half → float appears to be treated more strictly, resulting in ambiguity.</li>
<li>On GCC/Clang this may not trigger, which is why others may not have reported it.</li>
</ul>
<p>I resolved this problem with <code>static_cast&lt;float&gt;</code>, but it’s not a good enough solution for situations like this.</p>
<h3><a name="p-476255-better-solution-4" class="anchor" href="#p-476255-better-solution-4"></a>Better solution</h3>
<p>Following PyTorch conventions, we could use an <strong>accumulation type</strong> to ensure consistent promotion:</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">using acc_t = at::acc_type&lt;T, /*is_cuda=*/true&gt;;
acc_t v  = static_cast&lt;acc_t&gt;(offset_input[input_index]);
acc_t mv = static_cast&lt;acc_t&gt;(maxval);

if (v &gt; mv) {
    maxval = offset_input[input_index];
    maxidx = input_index;
}

</code></pre>
<p>And also initialize <code>maxval</code> in a type-safe way:</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">T maxval = is_empty ? T(0) : std::numeric_limits&lt;T&gt;::lowest();
</code></pre>
<h3><a name="p-476255-suggestion-5" class="anchor" href="#p-476255-suggestion-5"></a>Suggestion</h3>
<p>Would it make sense to update the ROI Pool kernel to use <code>acc_type</code> (and <code>lowest()</code>) to avoid this build-time ambiguity and preserve precision across float/double/half?</p>
<p>I can open a PR if maintainers think this is the right direction.</p>
<p>Please note that I’m still a junior-level software engineer, and this is my first time posting on the forum. I apologize in advance if my message is not fully aligned with the usual standards, and I appreciate your understanding.</p>

ptrblck: <p>Thank you for your description! I thinks it would be great if you could open an issue on GitHub describing your proposal there as well so the module maintainers could take a look and review it.</p>

kimchioverfit: <p>Thanks for your reply! I’ve opened an issue here: <a href="https://github.com/pytorch/vision/issues/9246" class="inline-onebox" rel="noopener nofollow ugc">[CUDA/MSVC][Suggestion] ROI Pool half-precision build error due to ambiguous comparison · Issue #9246 · pytorch/vision · GitHub</a></p>
<p>I’m not entirely sure if I submitted it in the correct way, but I thought it would be better to share it here first.</p>
<p>Really appreciate your suggestion and guidance!</p>