Contrastive Learning

lkm9983: <p>When it comes to contrastive learning, the objective is to maximize the similarity between similar data points while minimizing the similarity between dissimilar ones.</p>
<p>One of the commonly used contrastive losses is the NT-Xent loss, where “Sim” represents the cosine similarity between two data point representations.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee7f0139da9dc68a93f828811ac684c26483d736.png" data-download-href="https://discuss.pytorch.org/uploads/default/ee7f0139da9dc68a93f828811ac684c26483d736" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee7f0139da9dc68a93f828811ac684c26483d736_2_690x161.png" alt="image" data-base62-sha1="y1PK9KCOhFLPcIX39E1tpNysMYe" width="690" height="161" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee7f0139da9dc68a93f828811ac684c26483d736_2_690x161.png, https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee7f0139da9dc68a93f828811ac684c26483d736.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee7f0139da9dc68a93f828811ac684c26483d736.png 2x" data-dominant-color="F0F0F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">963×225 47.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>To minimize the loss, the numerator should be increasing, while the denominator should be decreasing. Since the denominator has “Sim” arguments, it has to decrease. The cosine similarity ranges from -1 to 1, and it seems that the closer the cosine similarity is to -1, the lower the loss becomes.</p>
<p>However, during my training process using this loss, I noticed that the similarity terms in the denominator converge to 0 instead of -1. (To be specific, sim(zi, zk) mentioned in the figure above)</p>
<p>Is there a mathematical explanation for this unexpected behavior?</p>

Joschka: <p>Without knowing the details on how you obtain your latents z and what code you use to compute their cosine similarity, it’s really hard to tell what exactly caused this behavior.</p>
<p>A wild guess would be that you actually monitor the similarity <em>after</em> applying the exponential. If your model converges such that 0 &lt; tau &lt;&lt; 1 and sim(zi, zk) is approx. equal to -1, 0 would be the expected outcome in that case.</p>

lkm9983: <p>Thanks for answering</p>
<p>Actually, Mine is SimCSE and tau is a hyperparameter(=0.05)<br>
(latent z can be obtained by passing through an encoder)</p>
<p>Sim(cosine_similarity) can get -1~1.<br>
The problem is that Sim(zi, zk) converges to 0 while sim(zi, zj), the positive pair, goes to 1 as expected<br>
And according to the loss formula, I think Sim should get close to -1.</p>
<p>I hope it makes sense.<br>
If you have any opinion and share it,  it would be very helpful to me.<br>
Thanks</p>

ANUMULA_CHAITANYA_SA: <p>is the problem solved ? I am curious to know the reason behind it</p>