Looking for PyTorch op data type promotion and result tensor data type rules

username: Sujoy_Saraswati
cooked: <p>Is there any document/guideline for PyTorch ops and data type conversions and promotion rules? There is a document at <a href="https://pytorch.org/docs/stable/tensor_attributes.html#type-promotion-doc" class="inline-onebox" rel="noopener nofollow ugc">Tensor Attributes — PyTorch 2.1 documentation</a>, however it does not cover some of the op specific cases. For example, add for int32 arguments returns an int32 dtype tensor, but div returns a float32 tensor -</p>
<blockquote>
<blockquote>
<blockquote>
<p>a = torch.ones(4, dtype=torch.int32)<br>
b = torch.ones(4, dtype=torch.int32)<br>
add = a + b<br>
add.dtype<br>
torch.int32<br>
div = a / b<br>
div.dtype<br>
torch.float32<br>
torch.result_type(a, b) ← This doesn’t return the correct result dtype for div<br>
torch.int32</p>
</blockquote>
</blockquote>
</blockquote>
<p>Similarly, for logical comparator ops like eq, the result tensor is boolean but for in-place eq_, the result tensor type is same as the original. Also, eq_ doesn’t seem to do a dtype conversion -</p>
<blockquote>
<blockquote>
<blockquote>
<p>ai.dtype<br>
torch.int32<br>
b.dtype<br>
torch.float32<br>
c = torch.eq(ai, b) ← this works fine<br>
ai.eq_(b) ← this doesn’t work<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
RuntimeError: Expected object of scalar type int but got scalar type float for argument ‘other’</p>
</blockquote>
</blockquote>
</blockquote>
<p>It would be great if anyone can share any document/guideline on the rules covering all these cases.</p>

username: wconstab
cooked: <p>Hey Sujoy,</p>
<p>A couple of helpful resources are:</p>
<p>the table that determines actual promotion behavior:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/master/c10/core/ScalarType.h#L422">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/master/c10/core/ScalarType.h#L422" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/c10/core/ScalarType.h#L422" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/master/c10/core/ScalarType.h#L422</a></h4>



    <pre class="onebox">      <code class="lang-h">
        <ol class="start lines" start="412" style="counter-reset: li-counter 411 ;">
            <li>      false,</li>
            <li>      "promoteTypes with quantized numbers is not handled yet; figure out what the correct rules should be, offending types: ",</li>
            <li>      toString(a),</li>
            <li>      " ",</li>
            <li>      toString(b));</li>
            <li>}</li>
            <li>
            </li>
<li>// this matrix has to be consistent with</li>
            <li>// AT_FORALL_SCALAR_TYPES_WITH_COMPLEX_AND_QINTS undefined is used where we</li>
            <li>// are not sure about the correct value for type promotion.</li>
            <li class="selected">static constexpr ScalarType _promoteTypesLookup[static_cast&lt;int&gt;(</li>
            <li>    ScalarType::NumOptions)][static_cast&lt;int&gt;(ScalarType::NumOptions)] = {</li>
            <li>    /*        u1  i1  i2  i4  i8  f2  f4  f8  c2  c4  c8  b1  q1  q2  q3  bf*/</li>
            <li>    /* u1 */ {u1, i2, i2, i4, i8, f2, f4, f8, c2, c4, c8, u1, ud, ud, ud, bf},</li>
            <li>    /* i1 */ {i2, i1, i2, i4, i8, f2, f4, f8, c2, c4, c8, i1, ud, ud, ud, bf},</li>
            <li>    /* i2 */ {i2, i2, i2, i4, i8, f2, f4, f8, c2, c4, c8, i2, ud, ud, ud, bf},</li>
            <li>    /* i4 */ {i4, i4, i4, i4, i8, f2, f4, f8, c2, c4, c8, i4, ud, ud, ud, bf},</li>
            <li>    /* i8 */ {i8, i8, i8, i8, i8, f2, f4, f8, c2, c4, c8, i8, ud, ud, ud, bf},</li>
            <li>    /* f2 */ {f2, f2, f2, f2, f2, f2, f4, f8, c2, c4, c8, f2, ud, ud, ud, f4},</li>
            <li>    /* f4 */ {f4, f4, f4, f4, f4, f4, f4, f8, c4, c4, c8, f4, ud, ud, ud, f4},</li>
            <li>    /* f8 */ {f8, f8, f8, f8, f8, f8, f8, f8, c8, c8, c8, f8, ud, ud, ud, f8},</li>
        </ol>
      </code>
    </pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>And a new ‘reference’ implementation in primtorch which should match:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/master/torch/_prims/utils.py#L676">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/master/torch/_prims/utils.py#L676" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/_prims/utils.py#L676" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/master/torch/_prims/utils.py#L676</a></h4>



    <pre class="onebox">      <code class="lang-py">
        <ol class="start lines" start="666" style="counter-reset: li-counter 665 ;">
            <li>class ELEMENTWISE_TYPE_PROMOTION_KIND(Enum):</li>
            <li>    DEFAULT = (0,)</li>
            <li>    INT_TO_FLOAT = (1,)</li>
            <li>    ALWAYS_BOOL = (2,)</li>
            <li>    OP_MATH = (3,)</li>
            <li>    COMPLEX_TO_FLOAT = (4,)</li>
            <li>    BOOL_TO_LONG = (5,)</li>
            <li>
            </li>
<li>
            </li>
<li># TODO: document type promotion kinds</li>
            <li class="selected">def elementwise_dtypes(</li>
            <li>    *_args, type_promotion_kind: ELEMENTWISE_TYPE_PROMOTION_KIND</li>
            <li>) -&gt; Tuple[torch.dtype, torch.dtype]:</li>
            <li>    """</li>
            <li>    Computes the computation and result dtypes for elementwise type promotion</li>
            <li>    on the given arguments and with the given elementwise type promotion kind.</li>
            <li>
            </li>
<li>    Note that not all inputs to an elementwise operation necessarily participate in type promotion.</li>
            <li>    For example, the "alpha" parameter of torch.add does not participate in type promotion,</li>
            <li>    although it is cast to the Python type corresponding to the computation dtype that</li>
            <li>    the type promotion algorithm determines.</li>
        </ol>
      </code>
    </pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Does this help?</p>