Repository: tensorflow/addons

Issue:
Implement Metric Losses
Per the [RFC](https://github.com/tensorflow/community/blob/393dfc9e03dd4150a839de6357a26e0259aa97d6/rfcs/20181214-move-to-addons.md), we need to move [metric_losses](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/losses/python/metric_learning) from contrib to addons:

- [x] pairwise_distance
- [x] contrastive_loss
- [x] triplet_semihard_loss
- [x] npairs_loss
- [x] npairs_loss_multilabel
- [x] lifted_struct_loss

This will involve inheriting from base Keras Loss, modifying the code to match those APIs, and modifying test cases to run in TF2.



Discussion:
Hi @seanpmorgan, from tensorflow testing meeting of 5th Feb, I liked the idea of saving useful parts from tf.contrib and add it to a add-on repo for tf-2.0 and would like to contribute to the same :). Can I contribute to this as I am not a gde and is this a good issue to start with
@tabshaikh Welcome, Tabish! I think, in all those functions, triplet_semihard_loss and lifted_struct_loss are easy to be implemented. I am in the process of moving triplet_semihard_loss (nearly finished), and I'll create a PR tomorrow if everything goes well (I'm on holiday). 

I think lifted_struct_loss is a good starter issue, would you like to take a try?  
@facaiy sure I would like to give lifted_struct_loss a try
Hello @seanpmorgan and @facaiy, I think I can take a look at the remaining losses. Is there anyone already responsible for these? If not, I'd like to give it a try. Thank you!
Welcome, @WindQAQ . I don't know anyone is working on them, so yes, just go head. 

By the way, I think #25 is easier for new contributor. Perhaps you are interested on it as well. Anyway, thanks for your help. Just ping us (or join https://gitter.im/tensorflow/sig-addons) if you need any help :-)
Okay. It's also free for me to deal with image functions first. Thanks for your quick reply :smiley:
Hey guys, because [all losses in addons](https://github.com/tensorflow/addons/tree/master/tensorflow_addons/losses) must inherit from `keras.losses.LossfunctionWrapper`, I have some API design issues here:

Since `fn`, which is passed into [LossFunctionWrapper](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/losses.py#L127), should be with two positional arguments, however, `contrastive_loss`, `npairs_loss` and `npairs_loss_multilabel` implemented in [contrib](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/losses/python/metric_learning/metric_loss_ops.py#L86) are with three arguments.

My initial thought is that (take `contrastive_loss` for example):

`embeddings_anchor` and `embeddings_positive` can be stacked to one tensor, and we can split it manually in addons implementation. Thus, the interface should be like
```python
def contrastive_loss(y_true, y_pred):
    embeddings_anchor = y_pred[0] # suppose they are stacked along with axis 0
    embeddings_positive = y_pred[1]
    # and so on

class ContrastiveLoss(LossFunctionWrapper):
    def __init__(self, name):
        super(ContrastiveLoss, self).__init__(fn=contrastive_loss, name=name)
```

One drawback here is that users should manipulate this stacked tensor on their own, which is not so intuitive though. Any feedback or thought would be really appreciated. Thanks!
@martinwicke @fchollet Could you give us some advice for @WindQAQ 's question? 
The `Loss` class fundamentally assumes that a signature with 3 tensor arguments, `y_true`, `y_pred`, and `sample_weight`. We can't go around it.

I think the right pattern here isn't a `Loss` subclass but a `Layer` subclass. [Here's how it would work](https://colab.sandbox.google.com/drive/1mz1vtC-iVAkgMQLRZGNBSDTUqTowgSNK).

```python
class ContrastiveLossLayer(keras.layers.Layer):

  def __init__(self, name=None):
    super(ContrastiveLossLayer, self).__init__(name=name)

  def __call__(self, y_true, positive, negative):
    # Pack all arguments in the first argument (optional, to expose a better __call__ signature)
    return super(LogisticHead, self).__call__([y_true, positive, negative])

  def call(self, inputs):
    y_true, positive, negative = inputs[0], inputs[1], inputs[2]
    loss = ...
    inference_time_predictions = ...
    self.add_loss(loss)
    return inference_time_predictions
```

Then to use it, you just insert it in your model as you would do for a regular layer (see the logistic loss layer example I linked).
Thanks for the suggestion! I do think this pattern is more flexible and beneficial to future extension.
@seanpmorgan and @facaiy, do you think addons can adopt this kind of design pattern instead of inheritance from `LossFunctionWrapper` as a unified interface. Or we can inherit from `LossFunctionWrapper` when it's possible but inherit from `Layer` when it comes to `ContrastiveLoss` case. Many thanks!
I'm fine with inheriting from `Layer`, but I'm not sure which module is appropriate for this kind of loss, `tfa.layer` or `tfa.loss`. @WindQAQ Could you pick one loss, say ContrastiveLossLayer, and write an example to show us how to use it with keras model?
@facaiy It should be something like siamese network. `ContrastiveLossLayer` can return other values such as euclidean distance (similarity) between two embeddings instead of loss. (Just a rough script here).
```python
import numpy as np

import tensorflow as tf


@tf.function
def contrastive_loss(y_true, embeddings_anchor, embeddings_positive,
                     margin=1.0):
    distances = tf.math.sqrt(
        tf.math.reduce_sum(
            tf.math.squared_difference(
                embeddings_anchor, embeddings_positive),
            1))

    return tf.math.reduce_mean(
        tf.cast(y_true, tf.dtypes.float32) * tf.math.square(distances) +
        (1. - tf.cast(y_true, tf.dtypes.float32)) *
        tf.math.square(tf.math.maximum(margin - distances, 0.)),
        name='contrastive_loss')


class ContrastiveLossLayer(tf.keras.layers.Layer):

    def __init__(self, margin=1.0, name=None):
        super(ContrastiveLossLayer, self).__init__(name=name)
        self._margin = margin

    def __call__(self, y_true, embeddings_anchor, embeddings_positive):
        return super(ContrastiveLossLayer, self).__call__([y_true, embeddings_anchor, embeddings_positive])

    def call(self, inputs):
        loss = contrastive_loss(*inputs, margin=self._margin)
        self.add_loss(loss)
        return loss


class L2Normalization(tf.keras.layers.Layer):

    def __init__(self, name=None):
        super(L2Normalization, self).__init__(name=name)

    def call(self, inputs):
        return tf.math.l2_normalize(inputs, axis=1)


def create_base_model(input_shape=(28, 28)):
    input = tf.keras.layers.Input(shape=input_shape)
    x = tf.keras.layers.Flatten()(input)
    x = tf.keras.layers.Dense(128, activation='relu')(x)
    x = L2Normalization()(x)
    return tf.keras.models.Model(input, x)


base_model = create_base_model()

input_a = tf.keras.layers.Input(shape=(28, 28), name="input_a")
input_b = tf.keras.layers.Input(shape=(28, 28), name="input_b")
labels = tf.keras.layers.Input(shape=(1,), name="labels")

output_a = base_model(input_a)
output_b = base_model(input_b)

outputs = ContrastiveLossLayer()(labels, output_a, output_b)

model = tf.keras.models.Model([input_a, input_b, labels], outputs=outputs)

model.compile(tf.keras.optimizers.Adam(1e-3))

model.summary()

fake_data = {
    'input_a': np.random.rand(1000, 28, 28),
    'input_b': np.random.rand(1000, 28, 28),
    'labels': np.random.randint(0, 2, size=(1000, 1)),
}

model.fit(fake_data, epochs=3)

```
@fchollet Thanks for your advice! I note that [add_loss](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/layers/Layer?hl=en#add_loss) 
> Note that add_loss is not supported when executing eagerly. 

Is there a plan to support eager mode in the near future?


@WindQAQ Thank you, Tzu-Wei. I'm kind of curious if the solution (use `layer.add_loss` and `model.compile(loss=None)` will works as we expected. Hence it could be much helpful if we could make a demo. If `ContrastiveLossLayer` is not a easy task,  would you minding creating a  MeanSquaredErrorLayer based on [MeanSquaredError](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/losses/MeanSquaredError?hl=en) ?

@seanpmorgan Sean, at the first glance at example provided by @WindQAQ , it seems that we should put `ContrastiveLossLayer` in the `tfa.layer` module. What do you think?
@facaiy Scripts are [here](https://gist.github.com/WindQAQ/70626d353145115b6c0e9977984e9938). I use MNIST and `sparse_categorical_crossentropy` as an example. Also, while I was implementing the scripts, I figured out some drawbacks if it's inherited from `Layer`:

1. The labels should be treated as one of the inputs. You can find this in both @fchollet and my script. I think this is not so good when it comes to inference time, during which, the ground truth labels are supposed to be missed.
2. Although we can use `self.add_metric` in `Layer` to trace some useful information such as accuracy, it seems that it's impossible for users to use `model.compile(..., metrics=['accuracy'])` to achieve the goal.

Therefore, I think inheritance from `Layer` is much more flexible for developers, but it's not an user-friendly API.
Nich work, Tzu-Wei, and thank you very much!

> The labels should be treated as one of the inputs.

Yes, that's the drawback what I thought in the morning. It looks kind of counter-intuitive for user.

I have a question after skimming [the paper](http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf
) .  Can we split `ContrastiveLoss` to two parts: 1, a layer to calculate `distance` between embeddings; 2, a loss `contrastive_loss(labels, distances)`?

![image](https://user-images.githubusercontent.com/1112263/54579656-7b16d600-4a3f-11e9-9f1b-1c970f46fcd0.png)


> I have a question after skimming [the paper](http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf) . Can we split `ContrastiveLoss` to two parts: 1, a layer to calculate `distance` between embeddings; 2, a loss `contrastive_loss(labels, distances)`?

I think this is probably the implementation in [keras example](https://github.com/keras-team/keras/blob/master/examples/mnist_siamese.py). In this case, `ContrastiveLoss` can exactly take two signature arguments after `embeddings_anchor` and `embeddings_positive` are pre-computed as distances. Not sure if this will meet requirements for addons' community.
Great, looks good to me. @seanpmorgan Sean, what do you think?
Similarly, in the cases of `npairs_loss` and `npairs_loss_multilabel`, I suppose that if leaving [L2 regularization terms](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/losses/python/metric_learning/metric_loss_ops.py#L276) behind, we can split it into two functions, say 

1. `compute_similarity_matrix(embeddings_anchor, embeddings_positive)` 
2. and `npairs_loss(labels, similarity_matrix)`.

Resolving patch:
diff --git a/tensorflow_addons/losses/__init__.py b/tensorflow_addons/losses/__init__.py
--- a/tensorflow_addons/losses/__init__.py
+++ b/tensorflow_addons/losses/__init__.py
@@ -21,6 +21,6 @@
 from tensorflow_addons.losses.contrastive import contrastive_loss, ContrastiveLoss
 from tensorflow_addons.losses.focal_loss import sigmoid_focal_crossentropy, SigmoidFocalCrossEntropy
 from tensorflow_addons.losses.lifted import lifted_struct_loss, LiftedStructLoss
-from tensorflow_addons.losses.npairs import npairs_loss, NpairsLoss
+from tensorflow_addons.losses.npairs import npairs_loss, NpairsLoss, npairs_multilabel_loss, NpairsMultilabelLoss
 from tensorflow_addons.losses.sparsemax_loss import sparsemax_loss, SparsemaxLoss
 from tensorflow_addons.losses.triplet import triplet_semihard_loss, TripletSemiHardLoss
diff --git a/tensorflow_addons/losses/npairs.py b/tensorflow_addons/losses/npairs.py
--- a/tensorflow_addons/losses/npairs.py
+++ b/tensorflow_addons/losses/npairs.py
@@ -64,6 +64,71 @@ def npairs_loss(y_true, y_pred):
     return tf.math.reduce_mean(loss)
 
 
+@keras_utils.register_keras_custom_object
+@tf.function
+def npairs_multilabel_loss(y_true, y_pred):
+    """Computes the npairs loss between multilabel data `y_true` and `y_pred`.
+
+    Npairs loss expects paired data where a pair is composed of samples from
+    the same labels and each pairs in the minibatch have different labels.
+    The loss takes each row of the pair-wise similarity matrix, `y_pred`,
+    as logits and the remapped multi-class labels, `y_true`, as labels.
+
+    To deal with multilabel inputs, the count of label intersection
+    is computed as follows:
+
+    ```
+    L_{i,j} = | set_of_labels_for(i) \cap set_of_labels_for(j) |
+    ```
+
+    Each row of the count based label matrix is further normalized so that
+    each row sums to one.
+
+    `y_true` should be a binary indicator for classes.
+    That is, if `y_true[i, j] = 1`, then `i`th sample is in `j`th class;
+    if `y_true[i, j] = 0`, then `i`th sample is not in `j`th class.
+
+    The similarity matrix `y_pred` between two embedding matrices `a` and `b`
+    with shape `[batch_size, hidden_size]` can be computed as follows:
+
+    ```python
+    # y_pred = a * b^T
+    y_pred = tf.matmul(a, b, transpose_a=False, transpose_b=True)
+    ```
+
+    See: http://www.nec-labs.com/uploads/images/Department-Images/MediaAnalytics/papers/nips16_npairmetriclearning.pdf
+
+    Args:
+      y_true: Either 2-D integer `Tensor` with shape
+        `[batch_size, num_classes]`, or `SparseTensor` with dense shape
+        `[batch_size, num_classes]`. If `y_true` is a `SparseTensor`, then
+        it will be converted to `Tensor` via `tf.sparse.to_dense` first.
+
+      y_pred: 2-D float `Tensor` with shape `[batch_size, batch_size]` of
+        similarity matrix between embedding matrices.
+
+    Returns:
+      npairs_multilabel_loss: float scalar.
+    """
+    y_pred = tf.convert_to_tensor(y_pred)
+    y_true = tf.cast(y_true, y_pred.dtype)
+
+    # Convert to dense tensor if `y_true` is a `SparseTensor`
+    if isinstance(y_true, tf.SparseTensor):
+        y_true = tf.sparse.to_dense(y_true)
+
+    # Enable efficient multiplication because y_true contains lots of zeros
+    # https://www.tensorflow.org/api_docs/python/tf/linalg/matmul
+    y_true = tf.linalg.matmul(
+        y_true, y_true, transpose_b=True, a_is_sparse=True, b_is_sparse=True)
+    y_true /= tf.math.reduce_sum(y_true, 1, keepdims=True)
+
+    loss = tf.nn.softmax_cross_entropy_with_logits(
+        logits=y_pred, labels=y_true)
+
+    return tf.math.reduce_mean(loss)
+
+
 @keras_utils.register_keras_custom_object
 class NpairsLoss(tf.keras.losses.Loss):
     """Computes the npairs loss between `y_true` and `y_pred`.
@@ -93,3 +158,48 @@ def __init__(self, name="npairs_loss"):
 
     def call(self, y_true, y_pred):
         return npairs_loss(y_true, y_pred)
+
+
+@keras_utils.register_keras_custom_object
+class NpairsMultilabelLoss(tf.keras.losses.Loss):
+    """Computes the npairs loss between multilabel data `y_true` and `y_pred`.
+
+    Npairs loss expects paired data where a pair is composed of samples from
+    the same labels and each pairs in the minibatch have different labels.
+    The loss takes each row of the pair-wise similarity matrix, `y_pred`,
+    as logits and the remapped multi-class labels, `y_true`, as labels.
+
+    To deal with multilabel inputs, the count of label intersection
+    is computed as follows:
+
+    ```
+    L_{i,j} = | set_of_labels_for(i) \cap set_of_labels_for(j) |
+    ```
+
+    Each row of the count based label matrix is further normalized so that
+    each row sums to one.
+
+    `y_true` should be a binary indicator for classes.
+    That is, if `y_true[i, j] = 1`, then `i`th sample is in `j`th class;
+    if `y_true[i, j] = 0`, then `i`th sample is not in `j`th class.
+
+    The similarity matrix `y_pred` between two embedding matrices `a` and `b`
+    with shape `[batch_size, hidden_size]` can be computed as follows:
+
+    ```python
+    # y_pred = a * b^T
+    y_pred = tf.matmul(a, b, transpose_a=False, transpose_b=True)
+    ```
+
+    See: http://www.nec-labs.com/uploads/images/Department-Images/MediaAnalytics/papers/nips16_npairmetriclearning.pdf
+
+    Args:
+      name: (Optional) name for the loss.
+    """
+
+    def __init__(self, name="npairs_multilabel_loss"):
+        super(NpairsMultilabelLoss, self).__init__(
+            reduction=tf.keras.losses.Reduction.NONE, name=name)
+
+    def call(self, y_true, y_pred):
+        return npairs_multilabel_loss(y_true, y_pred)


Test patch:
diff --git a/tensorflow_addons/losses/npairs_test.py b/tensorflow_addons/losses/npairs_test.py
--- a/tensorflow_addons/losses/npairs_test.py
+++ b/tensorflow_addons/losses/npairs_test.py
@@ -54,5 +54,84 @@ def test_unweighted(self):
         self.assertAllClose(loss, 0.253856)
 
 
+@test_utils.run_all_in_graph_and_eager_modes
+class NpairsMultilabelLossTest(tf.test.TestCase):
+    def config(self):
+        nml_obj = npairs.NpairsMultilabelLoss(name="nml")
+        self.assertEqual(nml_obj.name, "nml")
+        self.assertEqual(nml_obj.reduction, tf.keras.losses.Reduction.NONE)
+
+    def test_single_label(self):
+        """Test single label, which is the same with `NpairsLoss`."""
+        nml_obj = npairs.NpairsMultilabelLoss()
+        # batch size = 4, hidden size = 2
+        y_true = tf.constant(
+            [[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]],
+            dtype=tf.int64)
+        # features of anchors
+        f = tf.constant([[1., 1.], [1., -1.], [-1., 1.], [-1., -1.]],
+                        dtype=tf.float32)
+        # features of positive samples
+        fp = tf.constant([[1., 1.], [1., -1.], [-1., 1.], [-1., -1.]],
+                         dtype=tf.float32)
+        # similarity matrix
+        y_pred = tf.matmul(f, fp, transpose_a=False, transpose_b=True)
+        loss = nml_obj(y_true, y_pred)
+
+        # Loss = 1/4 * \sum_i log(1 + \sum_{j != i} exp(f_i*fp_j^T-f_i*f_i^T))
+        # Compute loss for i = 0, 1, 2, 3 without multiplier 1/4
+        # i = 0 => log(1 + sum([exp(-2), exp(-2), exp(-4)])) = 0.253846
+        # i = 1 => log(1 + sum([exp(-2), exp(-4), exp(-2)])) = 0.253846
+        # i = 2 => log(1 + sum([exp(-2), exp(-4), exp(-2)])) = 0.253846
+        # i = 3 => log(1 + sum([exp(-4), exp(-2), exp(-2)])) = 0.253846
+        # Loss = (0.253856 + 0.253856 + 0.253856 + 0.253856) / 4 = 0.253856
+
+        self.assertAllClose(loss, 0.253856)
+
+        # Test sparse tensor
+        y_true = tf.sparse.from_dense(y_true)
+        loss = nml_obj(y_true, y_pred)
+        self.assertAllClose(loss, 0.253856)
+
+    def test_multilabel(self):
+        nml_obj = npairs.NpairsMultilabelLoss()
+        # batch size = 4, hidden size = 2
+        y_true = tf.constant(
+            [[1, 1, 0, 0], [0, 1, 1, 0], [0, 0, 1, 1], [0, 0, 0, 1]],
+            dtype=tf.int64)
+        # features of anchors
+        f = tf.constant([[1., 1.], [1., -1.], [-1., 1.], [-1., -1.]],
+                        dtype=tf.float32)
+        # features of positive samples
+        fp = tf.constant([[1., 1.], [1., -1.], [-1., 1.], [-1., -1.]],
+                         dtype=tf.float32)
+        # similarity matrix
+        y_pred = tf.matmul(f, fp, transpose_a=False, transpose_b=True)
+        loss = nml_obj(y_true, y_pred)
+
+        # Loss = \sum_i log(1 + \sum_{j != i} exp(f_i*fp_j^T-f_i*f_i^T))
+        # Because of multilabel, the label matrix is normalized so that each
+        # row sums to one. That's why the multiplier before log exists.
+        # Compute loss for i = 0, 1, 2, 3 without multiplier 1/4
+        # i = 0 => 2/3 * log(1 + sum([exp(-2), exp(-2), exp(-4)])) +
+        #          1/3 * log(1 + sum([exp(2) , exp(0) , exp(-2)])) = 0.920522
+        # i = 1 => 1/4 * log(1 + sum([exp(2) , exp(-2), exp(0) ])) +
+        #          1/2 * log(1 + sum([exp(-2), exp(-4), exp(-2)])) +
+        #          1/4 * log(1 + sum([exp(2) , exp(4) , exp(2) ])) = 1.753856
+        # i = 2 => 1/4 * log(1 + sum([exp(2) , exp(4) , exp(2) ])) +
+        #          1/2 * log(1 + sum([exp(-2), exp(-4), exp(-2)])) +
+        #          1/4 * log(1 + sum([exp(0) , exp(-2), exp(2) ])) = 1.753856
+        # i = 4 => 1/2 * log(1 + sum([exp(-2), exp(0) , exp(2) ])) +
+        #          1/2 * log(1 + sum([exp(-4), exp(-2), exp(-2)])) = 1.253856
+        # Loss = (0.920522 + 1.753856 + 1.753856 + 1.253856) / 4 = 1.420522
+
+        self.assertAllClose(loss, 1.420522)
+
+        # Test sparse tensor
+        y_true = tf.sparse.from_dense(y_true)
+        loss = nml_obj(y_true, y_pred)
+        self.assertAllClose(loss, 1.420522)
+
+
 if __name__ == "__main__":
     tf.test.main()