Initialization error in transpose cnn

trm: <p>Hi all,</p>
<p>I was implementing transpose CNN from scratch . I noticed however pytorch seems to be adding a minor noise or some changes when it initializes .<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4eb0aad41860a0e9fffc0746b3212e695cfa545c.png" data-download-href="https://discuss.pytorch.org/uploads/default/4eb0aad41860a0e9fffc0746b3212e695cfa545c" title="weird pytorch"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4eb0aad41860a0e9fffc0746b3212e695cfa545c.png" alt="weird pytorch" data-base62-sha1="be7KxrPI1ybSVjWz8s6hUf3sIYQ" width="690" height="291" data-dominant-color="383838"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">weird pytorch</span><span class="informations">1008×426 39.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>At the top is my initialization in python for [1,1,3,3] kernel  and I copy paste it to pytorch transpose cnn layer using this command:</p>
<p>cnn.state_dict()[‘conv1_transpose.weight’].copy_(cnn_transpose_writeout, True)</p>
<p>In the image you will see the difference at the final digits. This actually does affect the results because they all add up and affect higher digits.</p>

ptrblck: <p>I think the difference comes from the usage of different <code>dtype</code>s. In the first print you can see that <code>float64</code> is returned while the second tensor does not print the <code>dtype</code> so would be using the default <code>float32</code> type, which has a reduced precision compared to <code>float64</code>.<br>
Which type of model are you using that <code>float64</code> is needed, as I haven’t seen such a use case before (besides some toy examples)?</p>

trm: <p>Its a toy example to compare an implementation.<br>
Everything is float 64 is in mine but I was curious if pytorch is doing things in float32.</p>
<p>Or perhaps it calls everything underneath in C and then maps it back to python. That communication layer between C and python might do some sort of compression. This is for the sake of increasing speeding of course</p>
<p>So in actual training performance is not affected (or maybe float64 is overkill)</p>
<p>The matrix below is the stored matrix in pytorch through the copy command. So after the copy it makes this minor modification.</p>

ptrblck: <aside class="quote no-group" data-username="trm" data-post="3" data-topic="133643">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/b5ac83/48.png" class="avatar"> trm:</div>
<blockquote>
<p>but I was curious if pytorch is doing things in float32.</p>
</blockquote>
</aside>
<p>Yes, by default, but you can also specify to use <code>float64</code> if needed.</p>
<aside class="quote no-group" data-username="trm" data-post="3" data-topic="133643">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/b5ac83/48.png" class="avatar"> trm:</div>
<blockquote>
<p>Or perhaps it calls everything underneath in C and then maps it back to python. That communication layer between C and python might do some sort of compression. This is for the sake of increasing speeding of course</p>
</blockquote>
</aside>
<p>No, there is no compression involved and PyTorch will use the specified dtype.</p>
<aside class="quote no-group" data-username="trm" data-post="3" data-topic="133643">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/b5ac83/48.png" class="avatar"> trm:</div>
<blockquote>
<p>So in actual training performance is not affected (or maybe float64 is overkill)</p>
</blockquote>
</aside>
<p>I haven’t seen a ML/DL model, which “needs” <code>float64</code> yet so the performance in the sense of model accuracy should not be affected. However, I would be interested to learn about such a model, which really needs 64 bits. The “speed” performance would be affected as you are not only using 2x the memory, but especially on the GPU would see a slowdown.</p>

trm: <p>Hi ,</p>
<p>I was building my own transpose cnn for scratch and I noticed a tiny difference between mine and pytorch at roughly 10^-7 digits onwards, so I was just suprised.</p>
<p>I have been writing my stuff on CPU so I did not take into GPU issues.</p>
<p>Its mostly a learning experience for me. Nothing to report but I would be interested to check if float 64 might reduce iterations a bit in the training process (in gpu this would be at the cost of computational speed and probably not worth doing).</p>