Bitcast pytorch vs tensorflow (different shapes?)

tyoc213: <p>Which implementation is “corrrect”??</p>
<h1><a name="going-from-tiny-to-big-1" class="anchor" href="#going-from-tiny-to-big-1"></a>going from tiny to big</h1>
<p>in pytorch</p>
<pre data-code-wrap="python"><code class="lang-python">torch.tensor([[0, 0, 3072, -16402], [0, 0, -25600, -16416], [0, 0, -14336, -16417], [0, 0, 30720, 16370]], dtype=torch.int16).view(torch.float64)
# outputs
tensor([[-0.9390],
        [-0.5190],
        [-0.4966],
        [ 1.1543]], dtype=torch.float64)
</code></pre>
<p>in tensorflow</p>
<pre data-code-wrap="python"><code class="lang-python">tensorflow.bitcast(tensorflow.constant([[0, 0, 3072, -16402], [0, 0, -25600, -16416], [0, 0, -14336, -16417], [0, 0, 30720, 16370]], dtype=tensorflow.float64), tensorflow.int8)
# outputs
[-0.938965,
    -0.519043,
    -0.496582,
    1.154297]
</code></pre>
<p>Data is same the difference is on dimention</p>
<ul>
<li>one less for tensorflow or one more for pytorch</li>
</ul>
<h1><a name="going-from-big-to-tinnier-2" class="anchor" href="#going-from-big-to-tinnier-2"></a>going from big to tinnier</h1>
<pre data-code-wrap="python"><code class="lang-python">torch.tensor([100000], dtype=torch.float64).view(torch.uint8)
tensor([0, 0, 0, 0, 0, 106, 248, 64], dtype=torch.uint8)
</code></pre>
<p>in tensorflow</p>
<pre data-code-wrap="python"><code class="lang-python">tensorflow.bitcast(tensorflow.constant([100000], dtype=tensorflow.float64), tensorflow.uint8)
# outputs
[[0, 0, 0, 0, 0, 106, 248, 64]]
</code></pre>
<p>Data is same the difference is on dimention</p>
<ul>
<li>one less for pytorch or one more for tensorflow</li>
</ul>
<p>They inverted shapes going from big to tiny or from tiny to big.</p>
<hr>
<p>I add this screenshots to show the problem</p>
<h1><a name="float64-to-uint8-3" class="anchor" href="#float64-to-uint8-3"></a>float64 to uint8</h1>
<ul>
<li><code>tensorflow.bitcast(tensorflow.constant([100000], dtype=tensorflow.float64), tensorflow.uint8)</code></li>
<li><code>torch.tensor([100000], dtype=torch.float64).view(torch.uint8)</code></li>
</ul>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/e/5eca828e968b064d8cd0ddc6072274221f97c5d2.png" alt="image" data-base62-sha1="dwyKZfRYq05PoollHx0hylcM1hg" width="534" height="65"><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c41aaae4fd6316c4d813958c5115f94c0d678489.png" data-download-href="https://discuss.pytorch.org/uploads/default/c41aaae4fd6316c4d813958c5115f94c0d678489" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c41aaae4fd6316c4d813958c5115f94c0d678489.png" alt="image" data-base62-sha1="rYOIAxOqlgim7tuOeAXbZ9IR7fj" width="690" height="55" data-dominant-color="353535"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">807×65 3.94 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<h1><a name="worse-this-one-differ-also-on-data-not-only-shape-4" class="anchor" href="#worse-this-one-differ-also-on-data-not-only-shape-4"></a>Worse, this one differ also on data, not only shape</h1>
<p>but numpy and pytorch match</p>
<ul>
<li><code>torch.tensor([[0, 0, 3072, -16402], [0, 0, -25600, -16416], [0, 0, -14336, -16417], [0, 0, 30720, 16370]], dtype=torch.float64).view(torch.int8)</code></li>
<li><code>tensorflow.bitcast(tensorflow.constant([[0, 0, 3072, -16402], [0, 0, -25600, -16416], [0, 0, -14336, -16417], [0, 0, 30720, 16370]], dtype=tensorflow.float64), tensorflow.int8)</code></li>
<li><code>np.array([[0, 0, 3072, -16402], [0, 0, -25600, -16416], [0, 0, -14336, -16417], [0, 0, 30720, 16370]], dtype=np.float64).view(dtype=np.int8)</code></li>
</ul>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/8/48195e6afaece333793b172b6bfc2c876a8a0661.png" data-download-href="https://discuss.pytorch.org/uploads/default/48195e6afaece333793b172b6bfc2c876a8a0661" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/8/48195e6afaece333793b172b6bfc2c876a8a0661.png" alt="image" data-base62-sha1="ahOJhli9PWXH458UXqfTXYhAZMt" width="690" height="231" data-dominant-color="232323"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1061×356 10.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/9/99a6e853c42836f300f211485e7913d63cb20707.png" data-download-href="https://discuss.pytorch.org/uploads/default/99a6e853c42836f300f211485e7913d63cb20707" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/9/99a6e853c42836f300f211485e7913d63cb20707.png" alt="image" data-base62-sha1="lVgElk4KDwvKuYxtmVrYG6D56sv" width="690" height="133" data-dominant-color="262728"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1109×214 20.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/a/5af5fbac099fe5e49481bf101de5ee95751d2a47.png" data-download-href="https://discuss.pytorch.org/uploads/default/5af5fbac099fe5e49481bf101de5ee95751d2a47" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/a/5af5fbac099fe5e49481bf101de5ee95751d2a47.png" alt="image" data-base62-sha1="cYFZiw7KZ9rqpHjFtQ9LS3pO1Yr" width="690" height="148" data-dominant-color="252525"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1079×233 8.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Why are you not using the same dtypes in both frameworks? E.g. I see <code>int8</code> and <code>int16</code>?</p>

tyoc213: <p>A yeah, is just a typo between all the test I did and copied here (fixed now), as I said, there is 
no difference in contents but in shapes
 difference in some cases in content too.</p>
<p>Look at the extra <code>[]</code> but they are symetric in the implementation of pytorch vs tensorflow.</p>
<p>Im more confused now… with the example that show not only different shape, but different values… (and to be clear by values I mean the whole row in the inner dimension, on the first capture only the outer shape is different, but on the second shape is different the inner values differ).</p>
<p>Does <code>bitcast</code> intensorflow and <code>.view(dtype)</code> in pytorch should be exactly equivalent? or they are implemented as wanted??? or Im doing something wrong that I cant see (apart from copy wrong example results)?</p>

ptrblck: <aside class="quote no-group" data-username="tyoc213" data-post="3" data-topic="206764">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tyoc213/48/3843_2.png" class="avatar"> tyoc213:</div>
<blockquote>
<p>Look at the extra <code>[]</code> but they are symetric in the implementation of pytorch vs tensorflow.</p>
</blockquote>
</aside>
<p>I don’t know how TF handles the shape, but it seems PyTorch keeps to numpy’s approach as a reference.</p>
<aside class="quote no-group" data-username="tyoc213" data-post="3" data-topic="206764">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tyoc213/48/3843_2.png" class="avatar"> tyoc213:</div>
<blockquote>
<p>Im more confused now… with the example that show not only different shape, but different values…</p>
</blockquote>
</aside>
<p>In case you are referring to the fist example, increase the precision during printing:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.set_printoptions(precision=6)
torch.tensor([[0, 0, 3072, -16402], [0, 0, -25600, -16416], [0, 0, -14336, -16417], [0, 0, 30720, 16370]], dtype=torch.int16).view(torch.float64)
# tensor([[-0.938965],
#         [-0.519043],
#         [-0.496582],
#         [ 1.154297]], dtype=torch.float64)
</code></pre>
<p>which matches the TF output.</p>
<p>Let me know if anything in PyTorch looks wrong (I’m not deeply familiar with TF, but would assume a bitcast should be equal).</p>