How is tensor storage size determined

username: ashari4
cooked: <p>In what cases does the underlying storage size not equal <code>t.numel() * t.element_size()</code>?</p>
<p>A known case is a tensor with all strides equal to 0.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; t = torch.empty_strided(size=(3,3),stride=(0,0),dtype=torch.float32)
&gt;&gt;&gt; t.storage().nbytes() 
4
</code></pre>

username: ashari4
cooked: <p>looks like the storage size of tensors created via <code>empty_strided</code> is given by “1 bigger than the offset of the last element according to stride”</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/051802e4f4fc6c03b653dd4552998354730fdef2/aten/src/ATen/EmptyTensor.cpp#L60">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/051802e4f4fc6c03b653dd4552998354730fdef2/aten/src/ATen/EmptyTensor.cpp#L60" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/051802e4f4fc6c03b653dd4552998354730fdef2/aten/src/ATen/EmptyTensor.cpp#L60" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/051802e4f4fc6c03b653dd4552998354730fdef2/aten/src/ATen/EmptyTensor.cpp#L60</a></h4>



    <pre class="onebox">      <code class="lang-cpp">
        <ol class="start lines" start="50" style="counter-reset: li-counter 49 ;">
            <li>}</li>
            <li>
            </li>
<li>size_t computeStorageNbytes(</li>
            <li>    IntArrayRef sizes,</li>
            <li>    IntArrayRef strides,</li>
            <li>    size_t itemsize_bytes,</li>
            <li>    size_t storage_offset</li>
            <li>  ) {</li>
            <li>  // Ignore overflow checks on mobile</li>
            <li>#ifndef C10_MOBILE</li>
            <li class="selected">  // size of the underlying storage is 1 bigger than the offset</li>
            <li>  // of the last element according to stride</li>
            <li>  uint64_t size = storage_offset + 1;</li>
            <li>  bool overflowed = false;</li>
            <li>  for (const auto i : c10::irange(sizes.size())) {</li>
            <li>    if (sizes[i] == 0) {</li>
            <li>      return 0;</li>
            <li>    }</li>
            <li>
            </li>
<li>    uint64_t strided_size;</li>
            <li>    overflowed |= c10::mul_overflows(strides[i], sizes[i] - 1, &amp;strided_size);</li>
        </ol>
      </code>
    </pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>And the size of a contiguous tensor and views of such tensors is given by the product along the dimensions</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/051802e4f4fc6c03b653dd4552998354730fdef2/aten/src/ATen/EmptyTensor.cpp#L39">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/051802e4f4fc6c03b653dd4552998354730fdef2/aten/src/ATen/EmptyTensor.cpp#L39" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/051802e4f4fc6c03b653dd4552998354730fdef2/aten/src/ATen/EmptyTensor.cpp#L39" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/051802e4f4fc6c03b653dd4552998354730fdef2/aten/src/ATen/EmptyTensor.cpp#L39</a></h4>



    <pre class="onebox">      <code class="lang-cpp">
        <ol class="start lines" start="29" style="counter-reset: li-counter 28 ;">
            <li>}  // namespace (anonymous)</li>
            <li>
            </li>
<li>size_t computeStorageNbytesContiguous(</li>
            <li>    IntArrayRef sizes,</li>
            <li>    size_t itemsize_bytes,</li>
            <li>    size_t storage_offset</li>
            <li>  ) {</li>
            <li>  // Ignore overflow checks on mobile</li>
            <li>#ifndef C10_MOBILE</li>
            <li>  uint64_t size = 1;</li>
            <li class="selected">  bool overflowed = c10::safe_multiplies_u64(sizes, &amp;size);</li>
            <li>  overflowed |= c10::add_overflows(size, storage_offset, &amp;size);</li>
            <li>  overflowed |= c10::mul_overflows(size, itemsize_bytes, &amp;size);</li>
            <li>  overflowed |= size &gt; storage_max();</li>
            <li>  TORCH_CHECK(!overflowed,</li>
            <li>              "Storage size calculation overflowed with sizes=", sizes);</li>
            <li>  return static_cast&lt;size_t&gt;(size);</li>
            <li>#else</li>
            <li>  const auto numel = c10::multiply_integers(sizes);</li>
            <li>  return itemsize_bytes * (storage_offset + numel);</li>
            <li>#endif</li>
        </ol>
      </code>
    </pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>