Problem with float precision

yegane: <p>I have several models, and I want to aggregate them.<br>
I want to compare two methods, that in one of them, I multiply a coefficient to model parameters, and in another I use torch.mean to aggregate.</p>
<p>I have a problem to get comparable results, and I think it is due the precision of float32 in model parameters.</p>
<p>e.g. if I have ten models, that I want to aggregate their models parameters, I could simply multiply them by 0.1 and then use torch.sum, and I expect to get the same result as torch.mean; yet I am not getting the same results. It is  my simplest scenario, but I have the same problem in more complex operations that I want to compare.</p>
<p>I tried to convert the elements of operations, which are in float32 to float64, and then do the operation and then convert them back to float32, but I get segmentation error.<br>
e.g. <code>(torch.DoubleTensor(a)*torch.DoubleTesnor(b)).float()</code><br>
Do you have any suggestion?</p>

ptrblck: <p>How large is the difference using <code>float32</code>?</p>
<p>Could you get a <code>gdb</code> backtrace for the segmentation fault?</p>

yegane: <p>difference is small, but in each round, it diverge more, and the final results are completely different, because it affect all model parameters, and they converge to different state. Therefore, the comparison of my methods is not reliable in other settings.</p>
<p>This is my first time with gdb, and I ran:</p>
<pre><code class="lang-auto">gdb 
&gt; file python
&gt; run myscript.py
</code></pre>
<p>I got the following results. If I should run it differently, please tell, but I got this:</p>
<pre><code class="lang-auto">[New Thread 0x7fff55fff700 (LWP 18812)]
[New Thread 0x7fff557fe700 (LWP 18813)]
[New Thread 0x7fff54ffd700 (LWP 18814)]

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
__strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:62
62      ../sysdeps/x86_64/multiarch/strlen-avx2.S: No such file or directory.
</code></pre>

KFrank: <p>Hello Yegane (and <a class="mention" href="/u/ptrblck">@ptrblck</a>)!</p>
<p>To add some pytorch version 0.3.0 color to this discussion, I can’t get<br>
this “use constructor to cast” approach to work either.  (But I get a more<br>
informative pytorch error, rather than a segmentation fault.)</p>
<p>(Note, that in terms of your original problem, I <em>can</em> use <code>.double()</code>,<br>
i.e., <code>(a.double() * b.double()).float())</code> works for me.)</p>
<p>Contrary to the (quite limited) documentation, I cannot construct<br>
a <code>DoubleTensor</code> from a <code>FloatTensor</code>, nor vice versa:</p>
<pre><code class="lang-python">import torch
torch.__version__

tf = torch.FloatTensor (2, 3)
type (tf)
td = torch.DoubleTensor (3, 4)
type (td)

torch.DoubleTensor (tf)
torch.FloatTensor (td)

tfd = tf.double()
type (tfd)
tdf = td.float()
type (tdf)
</code></pre>
<pre><code class="lang-nohighlight">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'0.3.0b0+591e73e'
&gt;&gt;&gt;
&gt;&gt;&gt; tf = torch.FloatTensor (2, 3)
&gt;&gt;&gt; type (tf)
&lt;class 'torch.FloatTensor'&gt;
&gt;&gt;&gt; td = torch.DoubleTensor (3, 4)
&gt;&gt;&gt; type (td)
&lt;class 'torch.DoubleTensor'&gt;
&gt;&gt;&gt;
&gt;&gt;&gt; torch.DoubleTensor (tf)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
TypeError: torch.DoubleTensor constructor received an invalid combination of arguments - got (torch.FloatTensor), but expected one of:
 * no arguments
 * (int ...)
      didn't match because some of the arguments have invalid types: (e[31;1mtorch.FloatTensore[0m)
 * (torch.DoubleTensor viewed_tensor)
      didn't match because some of the arguments have invalid types: (e[31;1mtorch.FloatTensore[0m)
 * (torch.Size size)
      didn't match because some of the arguments have invalid types: (e[31;1mtorch.FloatTensore[0m)
 * (torch.DoubleStorage data)
      didn't match because some of the arguments have invalid types: (e[31;1mtorch.FloatTensore[0m)
 * (Sequence data)
      didn't match because some of the arguments have invalid types: (e[31;1mtorch.FloatTensore[0m)

&gt;&gt;&gt; torch.FloatTensor (td)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
TypeError: torch.FloatTensor constructor received an invalid combination of arguments - got (torch.DoubleTensor), but expected one of:
 * no arguments
 * (int ...)
      didn't match because some of the arguments have invalid types: (e[31;1mtorch.DoubleTensore[0m)
 * (torch.FloatTensor viewed_tensor)
      didn't match because some of the arguments have invalid types: (e[31;1mtorch.DoubleTensore[0m)
 * (torch.Size size)
      didn't match because some of the arguments have invalid types: (e[31;1mtorch.DoubleTensore[0m)
 * (torch.FloatStorage data)
      didn't match because some of the arguments have invalid types: (e[31;1mtorch.DoubleTensore[0m)
 * (Sequence data)
      didn't match because some of the arguments have invalid types: (e[31;1mtorch.DoubleTensore[0m)

&gt;&gt;&gt;
&gt;&gt;&gt; tfd = tf.double()
&gt;&gt;&gt; type (tfd)
&lt;class 'torch.DoubleTensor'&gt;
&gt;&gt;&gt; tdf = td.float()
&gt;&gt;&gt; type (tdf)
&lt;class 'torch.FloatTensor'&gt;
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

ptrblck: <p>Thanks for the addition, <a class="mention" href="/u/kfrank">@KFrank</a>!<br>
The segfault might be related to <a href="https://github.com/pytorch/pytorch/issues/33899#issuecomment-592373737">this issue</a> and should be fixed per <a href="https://github.com/pytorch/pytorch/pull/34019">this PR</a>.</p>
<aside class="quote no-group" data-username="yegane" data-post="1" data-topic="79469">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yegane/48/10618_2.png" class="avatar"> yegane:</div>
<blockquote>
<p>e.g. if I have ten models, that I want to aggregate their models parameters, I could simply multiply them by 0.1 and then use torch.sum, and I expect to get the same result as torch.mean; yet I am not getting the same results. It is my simplest scenario, but I have the same problem in more complex operations that I want to compare.</p>
</blockquote>
</aside>
<p>You could try to force a specific order via loops, which might be slower, but would probably give you exactly the same results. Also, <code>float64</code> might be another good idea and I would try <a class="mention" href="/u/kfrank">@KFrank</a>’s suggestion on transforming the tensors.</p>