Multi-Similarity Loss. Multiple Similarities:
By: Keshav G
Tags: Deep Learning, Artificial Intelligence, Machine Learning, Programming, Technology

Metric learning aims to learn an embedding space, where the embedded vectors of similar samples are encouraged to be closer, while dissimilar ones are pushed apart from each other. Multi Similarity Loss proposed intuitively better methods to achieve this and is backed up by its accuracies across public benchmark datasets. This paper main contribution are two fold: a) Introducing multiple similarities into the mix, b) hard pair mining.

Multiple Similarities:

This loss deals with 3 types of similarities that carry the information of pairs.

1. Self Similarity:

x1 = anchor, x2 = positive, x3,x4 = negatives

Self similarity ensures that instances belonging to positive class remains closer to anchor than the instances associated with negative classes.

Sᵢₖ= cosine similarity between pairs, λ = similarity margin, α,β = hyperparameters

Ms-Loss comprises of two parts:

i) Positive Part:

x1 = Anchor, x2,x3 = positives, λ = margin

This part deals exclusively with p-pairs (positive pairs). λ represents the similarity margin which controls the closeness of p-pairs by heavily penalizing those p-pairs whose cosine similarity < λ. In the above diagram we can see two pairs x1-x2 and x1-x3, positive part of the loss for x1-x2 would be very low since e^(-α(Sᵢₖ — λ)) = e^(-α(0.7–0.5)) = e^(-0.2 * α), since α is hyperparameter and always greater than zero, value of this term will be very low when compared with positive part of the loss for x1-x3. For this pair, loss will be e^(-α(0.4–0.5)) = e^(0.1 α)). Cleⱼar distinction is there between the loss back propagated for x1-x2 and x1-x3.

ii) Negative Part:

x1 = anchor, x2,x3 = negatives, λ = margin

This part deals exclusively with n-pairs (negative pairs). This part of the loss ensures negatives to have as low as possible similarity with the anchor. This means that negatives lying closer to x1 ( i.e having high similarity) should be penalized heavily than negatives lying further away from x1 ( i.e having low similarity). This is evident from the loss, for eg. loss for x1-x2 is e^(β(Sᵢₖ — λ)) = e^(β(0.3–0.5)) = e^(-0.2 * β), whereas loss for x1-x3 is e^(β(0.1– 0.5)) = e^(-0.4 * β), since e^(-0.2 * β) > e^(-0.4 * β) for β > 0,hence loss for n-pairs with high similarity will be more than n-pairs with low similarity.

2. Negative relative similarity:

Weight assigned to a negative pair in ms loss, this is derived by derivative of the MS Loss with respect to a single pair.

Maths for the above eq. will be shared in another Blog. Weight wᵢⱼ for a pair is defined as contribution of loss from that pair towards the total loss.

weight for x1-x2, Case 1 > weight for x1-x2, Case 2> weight for x1-x2, Case 3

Taking into consideration only one n-pair x1-x2, ms-loss not only assign weight to this pair only on the basis of self-similarity between x1-x2 but also on the basis of its relative similarity i.e all other negatives present in the batch with respect to x1.

In the above eq. Sᵢⱼ refers to self similarity between x1-x2 and Sᵢₖ refers to similarity between x1-x3,x1-x4,x1-x5,x1-x6,x1-x7. In the above image though x1-x2 has same similarity Sᵢⱼ across all cases, its w-ᵢⱼ varies across cases.

Case 1: All other negatives (x3,x4,x5,x6,x7) are farther away from x1 with respect to x2.

Case 2: All negatives (x2,x3,x4,x5,x6,x7) lie at an equal distance from the anchor x1.

Case 3: All other negatives (x3,x4,x5,x6,x7) lie closer to x1 with respect to x2.

The difference in the w-ᵢⱼ of the three case is the denominator term Σ[e^(β(Sᵢₖ- Sᵢⱼ))]. Where Sᵢₖ = cosine similarity between x1-x3,x1-x4,x1-x5,x1-x6,x1-x7 and Sᵢⱼ =cosine similarity between x1-x2.

Case 1: w-ᵢⱼ is highest, since denominator term Σ[e^(β(Sᵢₖ- Sᵢⱼ))] is lowest as all Sᵢₖ<Sᵢⱼ making this e^(negative term)

Case 2: w-ᵢⱼ is in middle, since in denominator term Σ[e^(β(Sᵢₖ- Sᵢⱼ))], Sᵢₖ≃ Sᵢⱼ making it e^(zero-ish term).

Case 3: w-ᵢⱼ is lowest, since denominator term Σ[e^(β(Sᵢₖ- Sᵢⱼ))] is largest as all Sᵢₖ>Sᵢⱼ, therefore making this e^(positive term).

Case 3: e^(positive term) >Case 2: e^(zero-ish term) >Case 1: e^(negative term)

Since w-ᵢⱼ is inversely proportional to denominator term;

Case 3: w-ᵢⱼ >Case 2: w-ᵢⱼ>Case 1: w-ᵢⱼ

3. Positive relative similarity:

Weight assigned to a positive pair in ms loss, this is derived by derivative of the MS Loss with respect to a single pair.

weight for x1-x2, Case 1 > weight for x1-x2, Case 2> weight for x1-x2, Case 3

Negative relative similarity states the relationship of a single negative pair with all other negative pairs in the batch. Similarly positive relative similarity defines the relation between a single positive (x1-x2)and all other positives(x1-x3,x1-x4,x1-x5,x1-x6) in the batch. Following the procedure we did under Negative relative similarity heading, we can easily verify the results stated in the above image.

Mining Hard Positives And Negatives

Authors of Multi-Similarity loss paper used only hard negatives and positives for training and discarded all other pairs as they contribute little to no improvement and sometimes degraded the performance as well. Choosing only those pairs that carries the most information also make the algo computationally faster.

A = anchor, P = positives, N = negatives

i) Hard Negative Mining:

The above eq. states that only those negatives should be included in training whose similarity with the anchor is greater than minimum similarity of a positive (positive which is lying furthest in embedding space). Therefore in the above diagram only negatives we have chosen are the red ones since they are all lying inside the positive with minimum similarity to anchor, rest all negative are discarded.

ii) Hard Positive Mining:

The above eq. states that only those negatives should be included in training whose similarity with the anchor is less than maximum similarity of a negative (negative lying closest to anchor). Hard positives are coloured blue while rest are discarded.