nn.CosineSimilarity and custom cosine similarity using dot product giving different results

Megh_Bhalerao: <p>Hello,<br>
The following is a minimum working example of the problem that I have come across:</p>
<pre><code class="lang-auto">import torch 
import os
import numpy as np
import random
torch.use_deterministic_algorithms(True)
os.environ["CUBLAS_WORKSPACE_CONFIG"]=":16:8"
os.environ["CUBLAS_WORKSPACE_CONFIG"]=":4096:8"
import torch.nn as nn

seed = 0
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic =  True

x = torch.randn(28, device = "cuda", dtype=torch.float)
y = torch.randn(28, device = "cuda", dtype=torch.float)


my_dot = torch.dot(x, y)/torch.linalg.norm(y)
cos = nn.CosineSimilarity(dim = 0, eps = 0)

cos_dot = torch.linalg.norm(x) * cos(x,y)

print(my_dot.item())
print(cos_dot.item())
</code></pre>
<p>The output to this snippet on my system is the following -</p>
<pre><code class="lang-auto">0.15492278337478638
0.15492276847362518
</code></pre>
<p>They are different in the later decimal places, but both must be the same ideally.</p>
<p>When I cast <code>x</code> and <code>y</code> to double using the following lines instead of the above declaration like so -</p>
<pre><code class="lang-auto">x = torch.randn(28, device = "cuda", dtype=torch.float).double()
y = torch.randn(28, device = "cuda", dtype=torch.float).double()
</code></pre>
<p>I get the following outputs which are same and expected.</p>
<pre><code class="lang-auto">0.15492288182677755
0.15492288182677755
</code></pre>
<p>Why are they same in <code>double</code> precision but different in <code>float</code> precision?</p>
<p>Thanks!</p>

ptrblck: <p>The difference is ~1e-8 and is expected for <code>float32</code> due to the limited floating point precision and a potentially different order or operations.</p>

Megh_Bhalerao: <p>Thanks for the explanation, I understand. However, I have a follow up question, when I do the instantiation of <code>x</code> and <code>y</code> like the following, i.e. using <code>double</code> to initialize, I still get different answers for the cosine similarity calculated in 2 different ways - the code is the following, seeds are set same as above -</p>
<pre><code class="lang-auto">x = torch.randn(28, device = "cuda", dtype=torch.double)
y = torch.randn(28, device = "cuda", dtype=torch.double)

my_dot = torch.dot(x, y)/torch.linalg.norm(y)
cos = nn.CosineSimilarity(dim = 0, eps = 0)

cos_dot = torch.linalg.norm(x) * cos(x,y)

print(my_dot.item())
print(cos_dot.item())
</code></pre>
<p>The output of the above snippet is -</p>
<pre><code class="lang-auto">-0.139646650365121
-0.13964665036512092
</code></pre>
<p>I know that this is a small difference, but nevertheless it is causing my gradients (in my original code) to be nonzero which is causing my back propagation to diverge.</p>
<p>Please let me know if this is expected and if I am missing something.</p>
<p>Thanks again!</p>

ptrblck: <p>Increasing the bits in the numerical format will give you more precision (the new error is at ~1e-17) but will still be limited.<br>
I would suggest to check your actual requirement (negative gradients) and maybe to apply a small <code>eps</code> value to your calculation or so. You should not expect to get more precision that what’s possible in the current numerical format.</p>

Megh_Bhalerao: <p>What does numerical format exactly mean in the context of pytorch?</p>

ptrblck: <p>PyTorch uses <code>float32</code> (i.e. floating point numbers stored in 32 bits) as its default and allows users to use also wider types with more bits (and thus range and precision) such as <code>float64</code> as well as smaller types such as <code>float16</code> or <code>bfloat16</code>.<br>
E.g. take a look at <a href="https://en.wikipedia.org/wiki/Single-precision_floating-point_format">this Wikipedia article about <code>float32</code> which is also called “single-precision” float</a> for more general information about this format and the precision limitations.</p>
<p>The “precision” section might be interesting for you and you could play around with some information about the rounding behavior of this numerical format.<br>
E.g.:</p>
<blockquote>
<p>Precision limitations on integer values - Integers between <code>2**24</code> and <code>2**25</code> round to a multiple of 2 (even number)</p>
</blockquote>
<p>can be seen as:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.tensor(2**24, dtype=torch.float32)
print(x)
# tensor(16777216.)
print(x + 1)
# tensor(16777216.)
print(x + 2)
# tensor(16777218.)
</code></pre>
<p>As you can see, <code>16777217</code> is not representable in <code>float32</code> since the precision limits increase the larger the interval gets.</p>
<p>The round-off errors you are seeing are explained e.g. in <a href="https://en.wikipedia.org/wiki/Floating-point_arithmetic">this article</a> with a few examples.</p>