Why torch.nn.CosineSimilarity() gives different results for half and full tensor?

yxchng: <blockquote>
<p>import torch</p>
<p>x = torch.randn(10,2048).cuda()<br>
y = torch.randn(10,2048).cuda()</p>
<p>print(torch.nn.CosineSimilarity()(x,y).mean())</p>
<p>x= x.half()<br>
y =y.half()</p>
<p>print(torch.nn.CosineSimilarity()(x,y).mean())<br>
print(torch.diagonal(x @ y.T).mean())</p>
<p>x = torch.nn.functional.normalize(x)<br>
y = torch.nn.functional.normalize(y)</p>
<p>print(torch.nn.CosineSimilarity()(x,y).mean())<br>
print(torch.diagonal(x @ y.T).mean())</p>
</blockquote>
<p>gives</p>
<blockquote>
<p>tensor(-0.0104, device=‘cuda:0’)<br>
tensor(0., device=‘cuda:0’, dtype=torch.float16)<br>
tensor(-20.9531, device=‘cuda:0’, dtype=torch.float16)<br>
tensor(-0.0104, device=‘cuda:0’, dtype=torch.float16)<br>
tensor(-0.0104, device=‘cuda:0’, dtype=torch.float16)</p>
</blockquote>
<p>Normalizing the vectors give same result but that is not the expected behavior. There should not be any difference whether the vector is normalized or not normalized.</p>
<p>Is this a bug?</p>

ptrblck: <p><code>nn.CosineSimilarity</code> casts to <code>float32</code> using the mixed-precision utility via <code>torch.cuda.amp.autocast</code> as given <a href="https://pytorch.org/docs/stable/amp.html#ops-that-can-autocast-to-float32">here</a> so I assume the limited numerical precision of <code>float16</code> creates the large error. Since you are manually calling <code>half()</code> on the data <code>float16</code> will be used and ill thus create the mismatch.</p>

yxchng: <p>How can a switch to different data type create such a large error? It is just a simple dot product. If this kind of dot product can create such a large error with fp16, then it will should not be possible to train a neural network using fp16.</p>
<p>Doesn’t this indicate that there is something buggy here?</p>

ptrblck: <aside class="quote no-group" data-username="yxchng" data-post="4" data-topic="132283">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/7bcc69/48.png" class="avatar"> yxchng:</div>
<blockquote>
<p>How can a switch to different data type create such a large error?</p>
</blockquote>
</aside>
<p><code>float16</code> has a lower numerical range than <code>float32</code> and can thus easily over-/underflow.<br>
Have a look at <a href="https://en.wikipedia.org/wiki/Half-precision_floating-point_format"> Half-precision floating-point format - Wikipedia</a> to check the specifics and in particular:</p>
<blockquote>
<p>They can express values in the range ±65,504, with the minimum value above 1 being 1 + 1/1024.</p>
</blockquote>
<aside class="quote no-group" data-username="yxchng" data-post="4" data-topic="132283">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/7bcc69/48.png" class="avatar"> yxchng:</div>
<blockquote>
<p>It is just a simple dot product.</p>
</blockquote>
</aside>
<p>Which can easily overflow as seen if you manually apply <code>cosine_similarity</code> taken from <a href="https://github.com/pytorch/pytorch/blob/4a50b6c4905acc9c00597e7ff69a74591a8ef13e/aten/src/ATen/native/Distance.cpp#L242-L255">here</a>:</p>
<pre data-code-wrap="python"><code class="lang-python">def manual(x1_, x2_, dim, eps):
    # manual approach
    w12 = torch.sum(x1_ * x2_, dim)
    print(w12)
    w1 = torch.sum(x1_ * x1_, dim)
    print(w1)
    w2 = torch.sum(x2_ * x2_, dim)
    print(w2)
    out = (w1 * w2)
    print(out)
    out = out.clamp_min_(eps * eps)
    print(out)
    out = out.sqrt()
    print(out)
    n12 = out
    w12.div_(n12)
    print(w12.mean())

x = torch.randn(10,2048).cuda()
y = torch.randn(10,2048).cuda()
dim = 1
x1_ = x
x2_ = y
eps = 1e-8
manual(x1_, x2_, dim, eps)
print(torch.nn.CosineSimilarity()(x,y).mean())
manual(x1_.half(), x2_.half(), dim, eps)
</code></pre>
<p>You will see that the first call to <code>manual</code> using <code>float32</code> and <code>nn.CosineSimilarity</code> gives the same results, but are also using intermediate values outside of the valid <code>float16</code> range:</p>
<pre data-code-wrap="python"><code class="lang-python">tensor([  4.2845, -32.5558,   4.3298,   7.5555,  24.1382,  65.0509, -11.0637,
         80.8742,  -4.0695, -48.8819], device='cuda:0')
tensor([1968.8096, 2079.9971, 1968.8093, 2038.1968, 2118.2397, 2037.6740,
        2009.8625, 2059.7161, 1933.5315, 2092.7104], device='cuda:0')
tensor([2048.4983, 2051.1265, 2075.0337, 1997.3761, 2019.7664, 1949.5663,
        1937.5840, 2021.9969, 2035.3262, 2116.4380], device='cuda:0')
tensor([4033103.0000, 4266337.0000, 4085345.7500, 4071045.5000, 4278349.5000,
        3972580.5000, 3894277.5000, 4164739.5000, 3935367.2500, 4429092.0000],
       device='cuda:0')
tensor([4033103.0000, 4266337.0000, 4085345.7500, 4071045.5000, 4278349.5000,
        3972580.5000, 3894277.5000, 4164739.5000, 3935367.2500, 4429092.0000],
       device='cuda:0')
tensor([2008.2587, 2065.5112, 2021.2239, 2017.6832, 2068.4172, 1993.1333,
        1973.3923, 2040.7693, 1983.7760, 2104.5408], device='cuda:0')
tensor(0.0045, device='cuda:0')
tensor(0.0045, device='cuda:0')
</code></pre>
<p>In <code>float16</code> you would thus get invalid results:</p>
<pre data-code-wrap="python"><code class="lang-python">tensor([  4.2695, -32.5625,   4.3398,   7.5586,  24.1406,  65.0625, -11.0469,
         80.8750,  -4.0586, -48.9062], device='cuda:0', dtype=torch.float16)
tensor([1969., 2080., 1969., 2038., 2118., 2038., 2010., 2060., 1934., 2092.],
       device='cuda:0', dtype=torch.float16)
tensor([2048., 2052., 2076., 1997., 2020., 1950., 1938., 2022., 2035., 2116.],
       device='cuda:0', dtype=torch.float16)
tensor([inf, inf, inf, inf, inf, inf, inf, inf, inf, inf], device='cuda:0',
       dtype=torch.float16)
tensor([inf, inf, inf, inf, inf, inf, inf, inf, inf, inf], device='cuda:0',
       dtype=torch.float16)
tensor([inf, inf, inf, inf, inf, inf, inf, inf, inf, inf], device='cuda:0',
       dtype=torch.float16)
tensor(0., device='cuda:0', dtype=torch.float16)
</code></pre>
<aside class="quote no-group" data-username="yxchng" data-post="4" data-topic="132283">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/7bcc69/48.png" class="avatar"> yxchng:</div>
<blockquote>
<p>If this kind of dot product can create such a large error with fp16, then it will should not be possible to train a neural network using fp16.</p>
</blockquote>
</aside>
<p>No, I disagree as <code>torch.cuda.amp.autocast</code> is used exactly for this reason: operations prone to overflows or general numerical instability are kept in <code>float32</code>, while other operations are allowed to use <code>float16</code> (inputs and outputs) while the computation itself is often still done in <code>float32</code>.<br>
Since you are manually using <code>half()</code> on your data without depending on <code>autocast</code>, you would need to check the used operations and see if you might be running into numerical issues.</p>
<p>However, I agree that training models directly in <code>float16</code> is definitely not straightforward and might not work.</p>