Is there any way to set precision to be exact values instead of floats?

username: pytorcher
cooked: <p>I am trying to convert a program way back from pytorch 0.3 to 1.3.1.  I have it very close to doing the same thing when I debug batch by batch, but over time the floating point changes make it impossible to tell where my actual functions are different and where it is just floating point errors.  I know there is double() to make precision be higher, but that just changes where floating point errors happens.  Is there any mode that will still allow decimal points but give exact numbers and the rounding differences just get cut off.  Less precision would be fine for this as long as it exists in pytorch 0.3 and 1.3.1.</p>

username: KFrank
cooked: <p>Hi Py!</p>
<aside class="quote no-group" data-username="pytorcher" data-post="1" data-topic="82550" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/7ea924/48.png" class="avatar"> pytorcher:</div>
<blockquote>
<p>Is there any mode that will still allow decimal points but give exact numbers and the rounding differences just get cut off.</p>
</blockquote>
</aside>
<p>No.</p>
<p>There is neither a built-in way to do this, nor any practical way to<br>
do it “by hand.”</p>
<p>But you  speak as if you are convinced that your “actual functions<br>
are different,” even though the computational evidence you cited<br>
in your earlier thread shows that your “actual functions” agree up<br>
to floating-point round-off, which is as much as you can expect.</p>
<p>Floating-point round-off is real, and you can’t get rid of it.  But, in<br>
general, it’s not a problem.  After one iteration, your two versions<br>
agree up to floating-point round-off.  After a couple more iterations,<br>
the floating-point round-off accumulates some.  And after a few<br>
more iterations, it accumulates to the point that your optimizer’s<br>
trajectory in parameter space starts to wander off along two<br>
different paths.  But if you had run the same version twice, but<br>
where you initialized your model to slightly different random<br>
weights, your optimizer’s trajectory would also wander off along<br>
two different paths.</p>
<p>The loss surface in parameter space has a bunch of very<br>
complicated, high-dimensional peaks and valleys, and very<br>
many – for practical purposes equivalent – approximate minima.<br>
So it’s no surprise that slightly different initial conditions – or slightly<br>
different round-off error-- will set you off along different paths to<br>
different – but, in practice, equally good – approximate minima.</p>
<p>Without evidence that something is actually wrong, I think you’re<br>
seeing problems where there are none.</p>
<p>Best.</p>
<p>K. Frank</p>

username: pytorcher
cooked: <p>Hey K. Frank!</p>
<p>Thanks for responding again!  If you say there’s not a way to do this I’ll say this thread is solved and move back to the other one.  <a href="https://discuss.pytorch.org/t/how-to-debug-with-floating-point-differences/82397/6" class="inline-onebox">How to debug with floating point differences</a> (if anyone else is reading this later)</p>