Repository: tensorflow/addons

Issue:
FBetaScore is missing sample_weight parameter in update_state function
**System information**
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): see https://gist.github.com/PhilipMay/8897cd3980b4367f22cee7a0f8cb2ee5#file-poc_f1_bug-ipynb
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): macOS Mojave
- TensorFlow installed from (source or binary): binary (pypi with pip)
- TensorFlow version (use command below): tensorflow  2.0.0b1
- TensorFlow Addons installed from (source, PyPi): pypi
- TensorFlow Addons version: tfa-nightly 0.5.0.dev20190804
- Python version and type (eg. Anaconda Python, Stock Python as in Mac, or homebrew installed Python etc): conda python 3.6.9
- Bazel version (if compiling from source): -
- GCC/Compiler version (if compiling from source): -
- Is GPU used? (yes/no): no
- GPU model (if used): -

**Describe the bug**
Using `F1Score`from metrics with tf.keras does not work. See exception here:
https://gist.github.com/PhilipMay/8897cd3980b4367f22cee7a0f8cb2ee5#file-poc_f1_bug-ipynb

**Describe the expected behavior**
Should calculate the f1 metric

**Code to reproduce the issue**
see: https://gist.github.com/PhilipMay/8897cd3980b4367f22cee7a0f8cb2ee5#file-poc_f1_bug-ipynb

**Solution**
FBetaScore is missing sample_weight parameter in update_state function:
https://github.com/tensorflow/addons/blob/dab18121fa4253cf05320d41962bfbf1803e1f1b/tensorflow_addons/metrics/f_scores.py#L178

This would be a workaround for the problem:
`def update_state(self, y_true, y_pred, sample_weight=None):`

Although this way sample_weight is not considered in metric calculation.

Please note that all `update_state`functions from `tf.keras.metrics` do contain the `sample_weight` parameter: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/metrics.py

If you want me to I can prepare a PR.

Same problem should be here:
- https://github.com/tensorflow/addons/blob/master/tensorflow_addons/metrics/multilabel_confusion_matrix.py#L100
- https://github.com/tensorflow/addons/blob/master/tensorflow_addons/metrics/r_square.py#L54

**Other info / logs**
```
$ pip list
Package              Version             
-------------------- --------------------
absl-py              0.7.1               
appnope              0.1.0               
astor                0.8.0               
attrs                19.1.0              
backcall             0.1.0               
bleach               3.1.0               
certifi              2019.6.16           
cycler               0.10.0              
decorator            4.4.0               
defusedxml           0.6.0               
entrypoints          0.3                 
future               0.17.1              
gast                 0.2.2               
google-pasta         0.1.7               
grpcio               1.22.0              
h5py                 2.9.0               
hyperopt             0.1.2               
imbalanced-learn     0.5.0               
imblearn             0.0                 
ipykernel            5.1.1               
ipython              7.7.0               
ipython-genutils     0.2.0               
ipywidgets           7.5.1               
jedi                 0.14.1              
Jinja2               2.10.1              
joblib               0.13.2              
jsonschema           3.0.2               
jupyter              1.0.0               
jupyter-client       5.3.1               
jupyter-console      6.0.0               
jupyter-core         4.5.0               
Keras                2.2.4               
Keras-Applications   1.0.8               
keras-metrics        1.1.0               
Keras-Preprocessing  1.1.0               
kiwisolver           1.1.0               
lightgbm             2.2.3               
Markdown             3.1.1               
MarkupSafe           1.1.1               
matplotlib           3.1.1               
mistune              0.8.4               
nbconvert            5.5.0               
nbformat             4.4.0               
networkx             2.3                 
notebook             6.0.0               
numpy                1.17.0              
pandas               0.25.0              
pandocfilters        1.4.2               
parso                0.5.1               
pexpect              4.7.0               
pickleshare          0.7.5               
pip                  19.1.1              
prometheus-client    0.7.1               
prompt-toolkit       2.0.9               
protobuf             3.9.0               
ptyprocess           0.6.0               
Pygments             2.4.2               
pymongo              3.8.0               
pyparsing            2.4.2               
pyrsistent           0.15.4              
python-dateutil      2.8.0               
pytz                 2019.2              
PyYAML               5.1.2               
pyzmq                18.0.2              
qtconsole            4.5.2               
scikit-learn         0.21.3              
scipy                1.3.0               
Send2Trash           1.5.0               
setuptools           41.0.1              
six                  1.12.0              
tb-nightly           1.14.0a20190603     
tensorflow           2.0.0b1             
termcolor            1.1.0               
terminado            0.8.2               
testpath             0.4.2               
tf-estimator-nightly 1.14.0.dev2019060501
tfa-nightly          0.5.0.dev20190804   
tornado              6.0.3               
tqdm                 4.32.2              
traitlets            4.3.2               
wcwidth              0.1.7               
webencodings         0.5.1               
Werkzeug             0.15.5              
wheel                0.33.4              
widgetsnbextension   3.5.1               
wrapt                1.11.2              
xgboost              0.90    
```


Discussion:
cc @SSaishruthi @Squadrick 
I have this in my todo list. Will provide an update soon. 
@SSaishruthi great, thanks.


Resolving patch:
diff --git a/tensorflow_addons/metrics/f_scores.py b/tensorflow_addons/metrics/f_scores.py
--- a/tensorflow_addons/metrics/f_scores.py
+++ b/tensorflow_addons/metrics/f_scores.py
@@ -175,7 +175,9 @@ def __init__(self,
                 initializer='zeros',
                 dtype=self.dtype)
 
-    def update_state(self, y_true, y_pred):
+    # TODO: Add sample_weight support, currently it is
+    # ignored during calculations.
+    def update_state(self, y_true, y_pred, sample_weight=None):
         y_true = tf.cast(y_true, tf.int32)
         y_pred = tf.cast(y_pred, tf.int32)
 
@@ -338,6 +340,8 @@ def __init__(self, num_classes, average, name='f1_score',
         super(F1Score, self).__init__(
             num_classes, average, 1.0, name=name, dtype=dtype)
 
+    # TODO: Add sample_weight support, currently it is
+    # ignored during calculations.
     def get_config(self):
         base_config = super(F1Score, self).get_config()
         del base_config["beta"]


Test patch:
diff --git a/tensorflow_addons/metrics/f1_test.py b/tensorflow_addons/metrics/f1_test.py
--- a/tensorflow_addons/metrics/f1_test.py
+++ b/tensorflow_addons/metrics/f1_test.py
@@ -21,6 +21,8 @@
 import tensorflow as tf
 from tensorflow_addons.metrics import F1Score
 from tensorflow_addons.utils import test_utils
+from tensorflow.keras import layers
+import numpy as np
 
 
 @test_utils.run_all_in_graph_and_eager_modes
@@ -106,6 +108,23 @@ def test_f1_random_score_none(self):
         for avg, res in test_params:
             self._test_f1(avg, actuals, preds, res)
 
+    # Keras model check
+    def test_keras_model(self):
+        model = tf.keras.Sequential()
+        model.add(layers.Dense(64, activation='relu'))
+        model.add(layers.Dense(64, activation='relu'))
+        model.add(layers.Dense(1, activation='softmax'))
+        fb = F1Score(1, 'macro')
+        model.compile(
+            optimizer='rmsprop',
+            loss='categorical_crossentropy',
+            metrics=['acc', fb])
+        # data preparation
+        data = np.random.random((10, 3))
+        labels = np.random.random((10, 1))
+        labels = np.where(labels > 0.5, 1, 0)
+        model.fit(data, labels, epochs=1, batch_size=32, verbose=0)
+
 
 if __name__ == '__main__':
     tf.test.main()
diff --git a/tensorflow_addons/metrics/fbeta_test.py b/tensorflow_addons/metrics/fbeta_test.py
--- a/tensorflow_addons/metrics/fbeta_test.py
+++ b/tensorflow_addons/metrics/fbeta_test.py
@@ -21,6 +21,8 @@
 import tensorflow as tf
 from tensorflow_addons.metrics import FBetaScore
 from tensorflow_addons.utils import test_utils
+from tensorflow.keras import layers
+import numpy as np
 
 
 @test_utils.run_all_in_graph_and_eager_modes
@@ -112,6 +114,23 @@ def test_fbeta_random_score_none(self):
         for beta, res in test_params:
             self._test_fbeta(None, beta, actuals, preds, res)
 
+    # Keras model check
+    def test_keras_model(self):
+        model = tf.keras.Sequential()
+        model.add(layers.Dense(64, activation='relu'))
+        model.add(layers.Dense(64, activation='relu'))
+        model.add(layers.Dense(1, activation='softmax'))
+        fb = FBetaScore(1, 'macro')
+        model.compile(
+            optimizer='rmsprop',
+            loss='categorical_crossentropy',
+            metrics=['acc', fb])
+        # data preparation
+        data = np.random.random((10, 3))
+        labels = np.random.random((10, 1))
+        labels = np.where(labels > 0.5, 1, 0)
+        model.fit(data, labels, epochs=1, batch_size=32, verbose=0)
+
 
 if __name__ == '__main__':
     tf.test.main()