What is aten::fill_ meaning in ATen APIs?

fishmingyu: <p>I profile a model using the torch.profiler, only to find I am confused with the aten::fill_ API. Since there is no specific documentation about ATen, I couldn’t make sense of the original code in the following URL.</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/87465a6e68b62fe9a10482b98f42b48c71aacb68/aten/src/ATen/native/Fill.cpp#L35">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/87465a6e68b62fe9a10482b98f42b48c71aacb68/aten/src/ATen/native/Fill.cpp#L35" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/87465a6e68b62fe9a10482b98f42b48c71aacb68/aten/src/ATen/native/Fill.cpp#L35" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/87465a6e68b62fe9a10482b98f42b48c71aacb68/aten/src/ATen/native/Fill.cpp#L35</a></h4>



  <pre class="onebox">    <code class="lang-cpp">
      <ol class="start lines" start="25" style="counter-reset: li-counter 24 ;">
          <li>  auto iter = TensorIteratorConfig()</li>
          <li>    .set_check_mem_overlap(false)  // Fill is idempotent, so overlap is okay</li>
          <li>    .check_all_same_dtype(false)</li>
          <li>    .add_output(self)</li>
          <li>    .resize_outputs(false)</li>
          <li>    .build();</li>
          <li>  fill_stub(iter.device_type(), iter, value);</li>
          <li>  return self;</li>
          <li>}</li>
          <li>
          </li>
<li class="selected">Tensor&amp; fill_(Tensor&amp; self, const Scalar&amp; value) {</li>
          <li>  return fill_out(self, value);</li>
          <li>}</li>
          <li>
          </li>
<li>Tensor&amp; fill_(Tensor&amp; self, const Tensor&amp; value) {</li>
          <li>  TORCH_CHECK(value.dim() == 0, "fill_ only supports 0-dimension value tensor but got tensor with ", value.dim(), " dimensions.");</li>
          <li>  return fill_out(self, value.item());</li>
          <li>}</li>
          <li>
          </li>
<li>Tensor&amp; fill_meta_(Tensor&amp; self, const Scalar&amp; value) {</li>
          <li>  return self;</li>
      </ol>
    </code>
  </pre>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>What’s the difference between aten::fill_ on GPU and CPU? Compared to GPU, it seems that aten::fill_ costs much more time while running the same model on CPU.</p>
<p>If anyone is familiar with ATen and its APIs, please show me the points of their working mechanism.</p>

eqy: <p><code>fill_</code> looks to be a typical TensorIterator (roughly summarized, a highly templated way to implement elmentwise operations) op <a href="https://github.com/pytorch/pytorch/blob/a9b0a921d592b328e7e80a436ef065dadda5f01b/aten/src/ATen/native/Fill.cpp#L14" class="inline-onebox" rel="noopener nofollow ugc">pytorch/Fill.cpp at a9b0a921d592b328e7e80a436ef065dadda5f01b · pytorch/pytorch · GitHub</a>. TensorIterator ops are usually very generic so there likely isn’t anything that stands out vs. other elementwise ops.</p>
<p>Can you describe how you are timing the relative time cost of <code>fill_</code> on CPU and GPU? It is expected that the first invocation of fill may be slow depending on when timing starts/how the CUDA context is created, but I would be surprised if it remains slower when filling large tensors (due to the typical difference in GPU/CPU memory bandwidth).</p>