Repository: tensorflow/addons

Issue:
Bug in F1 and FBeta implementation and test
**Describe the bug**
The tests for FBeta and F1 both use a softmax function with just one output. 
- https://github.com/tensorflow/addons/blob/76249547c481ed47bd6f2f8092a99cd64d37fd2f/tensorflow_addons/metrics/fbeta_test.py#L122
- https://github.com/tensorflow/addons/blob/76249547c481ed47bd6f2f8092a99cd64d37fd2f/tensorflow_addons/metrics/f1_test.py#L116

The effect is that the output (prediction) is always 1. The right activation function for a binary calssification (with just one output is sigmoid). But this is not the only problem. This bug in the test hides an even worse other bug.

The other bug is that F1 and FBeta both can not handle values other then 1 (1.0) and 0 (0.0) as the predicted result. But the predictions of a binary classification (with simoid) and multi class classification with softmax is always somewhere between 1 and 0 (one hot encoded for multi class). The result of this bug is that this implementation of F1 and FBeta always return 0.0 when the predicted results are not exactly 0 (0.0) or 1 (1.0) - which is not realistic.

Also all other tests of F1 and FBeta have values of 0 or 1 as the predicted results. This does not reflect reality.
- https://github.com/tensorflow/addons/blob/76249547c481ed47bd6f2f8092a99cd64d37fd2f/tensorflow_addons/metrics/fbeta_test.py#L91
- https://github.com/tensorflow/addons/blob/76249547c481ed47bd6f2f8092a99cd64d37fd2f/tensorflow_addons/metrics/f1_test.py#L88

**Code to reproduce the issue**
Just change softmax to sigmoid in the tests and change `verbose=1` and see the results.



Discussion:
I guess something like this needs to be added:
https://github.com/tensorflow/tensorflow/blob/2ff39d00faf8f7e433ddcae0aa278f6e573b0c55/tensorflow/python/keras/metrics.py#L2767
Thanks, @PhilipMay. You are right that result will not be an exact number. 
The initial implementation we had do not indicate the probability. 

`preds = tf.constant([[0, 0, 1], [1, 1, 0], [1, 1, 1]], dtype=tf.int32) `

In this [0, 0, 1] does not indicate probability. Think we have 3 classes and the predicted is class 3 which is encoded as [0,0,1]

We may need to add argmax to change that in case if we are feeding directly.

Will take up this one and post the updates 
> We may need to add argmax to change that in case if we are feeding directly.

Yes - if we want to be able to add the metric to keras at training time we need that. Mybe you could have a look at the metrics implementations in Tensorflow.

Thanks you very much. :-)
@seanpmorgan Can you please assign this issue to me?

This here seems to work for me - not sure if it still has bugs in border cases or can be implemented more efficient:
``` python
    def update_state(self, y_true, y_pred, sample_weight=None):
        cond = tf.equal(y_pred, tf.reduce_max(y_pred))
        y_pred = tf.where(cond, y_pred, tf.zeros_like(y_pred))
        not_cond = tf.math.logical_not(cond)
        y_pred = tf.where(not_cond, y_pred, tf.ones_like(y_pred))
        
        y_true = tf.cast(y_true, tf.int32)
        y_pred = tf.cast(y_pred, tf.int32)
```
For further tests I would suggest that you can compare your results with those from sklearn:
``` python
from sklearn.metrics import f1_score
ytrue = np.argmax(y_test, axis=-1)
ypred = np.argmax(y_hat_test, axis=-1)

print(f1_score(ytrue, ypred, average='macro'))  
print(f1_score(ytrue, ypred, average='micro'))  
print(f1_score(ytrue, ypred, average='weighted'))
print(f1_score(ytrue, ypred, average=None))
```
Thanks @PhilipMay 
I have a working code ready now using threshold option. That will help in both multiclass and multilabel. Will create a PR today
> Thanks @PhilipMay
> I have a working code ready now using threshold option. That will help in both multiclass and multilabel. Will create a PR today

That is great. Thanks. I need that fix for my project. :-)
Maybe you can add a comparison with f1_score from sklearn to the tests.
@SSaishruthi I do not think that this implementation with threshold fixes the problem. Here is why:

When you have a "single-label categorial classification" where one sample belongs to exactly one class of many possible classes you apply a softmax funcation. Let me give an example:

You habe 3 classes: dog, cat and bike and do a one hot encoding.
```
dog = [1, 0, 0]
cat = [0, 1, 0]
bike = [0, 0, 1]
```

Now when you put a dog into the model you might get the following from the softmax function:
`result = [0.5, 0.25, 0.25]`
This means: dog because 0.5 is the highest value. And this is the reason why a threshold is not valid. You should apply something like `reduce_max` or `argmax`. Something like this: https://github.com/tensorflow/addons/issues/490#issuecomment-529882338
@PhilipMay 

I am taking binary accuracy as a reference here: https://github.com/tensorflow/tensorflow/blob/2ff39d00faf8f7e433ddcae0aa278f6e573b0c55/tensorflow/python/keras/metrics.py#L630

In order for this to work you need to provide the threshold value as 0.49 above. I can probably fix the threshold default to be 0.5. I think below 0.5, it may not be considered as a good prediction.


I have this tested with scikit learn as (that's the usual procedure I follow) 
well.https://colab.research.google.com/drive/1qSq0SsYkPqjdKUgM1RM4kKM67X75ocFj

Goal here is to make it compatible with both multi-class and multi-label

Resolving patch:
diff --git a/tensorflow_addons/metrics/f_scores.py b/tensorflow_addons/metrics/f_scores.py
--- a/tensorflow_addons/metrics/f_scores.py
+++ b/tensorflow_addons/metrics/f_scores.py
@@ -12,220 +12,155 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 # ==============================================================================
-"""Implements F1 scores."""
+"""Implements F scores."""
 
 from __future__ import absolute_import
 from __future__ import division
 from __future__ import print_function
 
 import tensorflow as tf
-from tensorflow.keras.metrics import Metric
-import numpy as np
+from tensorflow_addons.utils import keras_utils
 
 
-class FBetaScore(Metric):
-    """Computes F-Beta Score.
+@keras_utils.register_keras_custom_object
+class FBetaScore(tf.keras.metrics.Metric):
+    """Computes F-Beta score.
 
-    This is the weighted harmonic mean of precision and recall.
-    Output range is [0, 1].
+    It is the weighted harmonic mean of precision
+    and recall. Output range is [0, 1]. Works for
+    both multi-class and multi-label classification.
 
-    F-Beta = (1 + beta^2) * ((precision * recall) /
-                             ((beta^2 * precision) + recall))
-
-    `beta` parameter determines the weight given to the
-    precision and recall.
-
-    `beta < 1` gives more weight to the precision.
-    `beta > 1` gives more weight to the recall.
-    `beta == 1` gives equal weight to precision and recall.
+    F-Beta = (1 + beta^2) * (prec * recall) / ((beta^2 * prec) + recall)
 
     Args:
-       num_classes : Number of unique classes in the dataset.
-       average : Type of averaging to be performed on data.
-                   Acceptable values are None, micro, macro and
-                   weighted.
-       beta : float. Determines the weight of precision and recall
-                in harmonic mean. Acceptable values are either a number
-                of float data type greater than 0.0 or a scale tensor
-                of dtype tf.float32.
+        num_classes: Number of unique classes in the dataset.
+        average: Type of averaging to be performed on data.
+            Acceptable values are `None`, `micro`, `macro` and
+            `weighted`. Default value is None.
+        beta: Determines the weight of precision and recall
+            in harmonic mean. Determines the weight given to the
+            precision and recall. Default value is 1.
+        threshold: Elements of `y_pred` greater than threshold are
+            converted to be 1, and the rest 0. If threshold is
+            None, the argmax is converted to 1, and the rest 0.
 
     Returns:
-       F Beta Score: float
+        F-Beta Score: float
 
     Raises:
-       ValueError: If the `average` has values other than
-       [None, micro, macro, weighted].
+        ValueError: If the `average` has values other than
+        [None, micro, macro, weighted].
 
-       ValueError: If the `beta` value is less than or equal
-       to 0.
+        ValueError: If the `beta` value is less than or equal
+        to 0.
 
     `average` parameter behavior:
+        None: Scores for each class are returned
+
+        micro: True positivies, false positives and
+            false negatives are computed globally.
+
+        macro: True positivies, false positives and
+            false negatives are computed for each class
+            and their unweighted mean is returned.
 
-    1. If `None` is specified as an input, scores for each
-       class are returned.
-
-    2. If `micro` is specified, metrics like true positivies,
-       false positives and false negatives are computed
-       globally.
-
-    3. If `macro` is specified, metrics like true positivies,
-       false positives and false negatives are computed for
-       each class and their unweighted mean is returned.
-       Imbalance in dataset is not taken into account for
-       calculating the score
-
-    4. If `weighted` is specified, metrics are computed for
-       each class and returns the mean weighted by the
-       number of true instances in each class taking data
-       imbalance into account.
-
-    Usage:
-    ```python
-    actuals = tf.constant([[1, 1, 0],[1, 0, 0]],
-              dtype=tf.int32)
-    preds = tf.constant([[1, 0, 0],[1, 0, 1]],
-             dtype=tf.int32)
-    # F-Beta Micro
-    fb_score = tfa.metrics.FBetaScore(num_classes=3,
-                beta=0.4, average='micro')
-    fb_score.update_state(actuals, preds)
-    print('F1-Beta Score is: ',
-           fb_score.result().numpy()) # 0.6666666
-    # F-Beta Macro
-    fb_score = tfa.metrics.FBetaScore(num_classes=3,
-           beta=0.4, average='macro')
-    fb_score.update_state(actuals, preds)
-    print('F1-Beta Score is: ',
-          fb_score.result().numpy()) # 0.33333334
-    # F-Beta Weighted
-    fb_score = tfa.metrics.FBetaScore(num_classes=3,
-               beta=0.4, average='weighted')
-    fb_score.update_state(actuals, preds)
-    print('F1-Beta Score is: ',
-          fb_score.result().numpy()) # 0.6666667
-    # F-Beta score for each class (average=None).
-    fb_score = tfa.metrics.FBetaScore(num_classes=3,
-               beta=0.4, average=None)
-    fb_score.update_state(actuals, preds)
-    print('F1-Beta Score is: ',
-         fb_score.result().numpy()) # [1. 0. 0.]
-    ```
+        weighted: Metrics are computed for each class
+            and returns the mean weighted by the
+            number of true instances in each class.
     """
 
     def __init__(self,
                  num_classes,
                  average=None,
                  beta=1.0,
+                 threshold=None,
                  name='fbeta_score',
                  dtype=tf.float32):
         super(FBetaScore, self).__init__(name=name)
-        self.num_classes = num_classes
-        # type check
-        if not isinstance(beta, float) and beta.dtype != tf.float32:
-            raise TypeError("The value of beta should be float")
-        # value check
-        if beta <= 0.0:
-            raise ValueError("beta value should be greater than zero")
-        else:
-            self.beta = beta
+
         if average not in (None, 'micro', 'macro', 'weighted'):
             raise ValueError("Unknown average type. Acceptable values "
                              "are: [None, micro, macro, weighted]")
-        else:
-            self.average = average
-            if self.average == 'micro':
-                self.axis = None
-            else:
-                self.axis = 0
-        if self.average == 'micro':
-            self.true_positives = self.add_weight(
-                'true_positives',
-                shape=[],
-                initializer='zeros',
-                dtype=self.dtype)
-            self.false_positives = self.add_weight(
-                'false_positives',
-                shape=[],
-                initializer='zeros',
-                dtype=self.dtype)
-            self.false_negatives = self.add_weight(
-                'false_negatives',
-                shape=[],
-                initializer='zeros',
-                dtype=self.dtype)
-        else:
-            self.true_positives = self.add_weight(
-                'true_positives',
-                shape=[self.num_classes],
-                initializer='zeros',
-                dtype=self.dtype)
-            self.false_positives = self.add_weight(
-                'false_positives',
-                shape=[self.num_classes],
-                initializer='zeros',
-                dtype=self.dtype)
-            self.false_negatives = self.add_weight(
-                'false_negatives',
-                shape=[self.num_classes],
-                initializer='zeros',
-                dtype=self.dtype)
-            self.weights_intermediate = self.add_weight(
-                'weights',
-                shape=[self.num_classes],
+
+        if not isinstance(beta, float):
+            raise TypeError("The value of beta should be a python float")
+
+        if beta <= 0.0:
+            raise ValueError("beta value should be greater than zero")
+
+        if threshold is not None:
+            if not isinstance(threshold, float):
+                raise TypeError(
+                    "The value of threshold should be a python float")
+            if threshold > 1.0 or threshold <= 0.0:
+                raise ValueError("threshold should be between 0 and 1")
+
+        self.num_classes = num_classes
+        self.average = average
+        self.beta = beta
+        self.threshold = threshold
+        self.axis = None
+        self.init_shape = []
+
+        if self.average != 'micro':
+            self.axis = 0
+            self.init_shape = [self.num_classes]
+
+        def _zero_wt_init(name):
+            return self.add_weight(
+                name,
+                shape=self.init_shape,
                 initializer='zeros',
                 dtype=self.dtype)
 
+        self.true_positives = _zero_wt_init('true_positives')
+        self.false_positives = _zero_wt_init('false_positives')
+        self.false_negatives = _zero_wt_init('false_negatives')
+        self.weights_intermediate = _zero_wt_init('weights_intermediate')
+
     # TODO: Add sample_weight support, currently it is
     # ignored during calculations.
     def update_state(self, y_true, y_pred, sample_weight=None):
+        if self.threshold is None:
+            threshold = tf.reduce_max(y_pred, axis=-1, keepdims=True)
+            # make sure [0, 0, 0] doesn't become [1, 1, 1]
+            # Use abs(x) > eps, instead of x != 0 to check for zero
+            y_pred = tf.logical_and(y_pred >= threshold,
+                                    tf.abs(y_pred) > 1e-12)
+        else:
+            y_pred = y_pred > self.threshold
+
         y_true = tf.cast(y_true, tf.int32)
         y_pred = tf.cast(y_pred, tf.int32)
 
-        # true positive
-        self.true_positives.assign_add(
-            tf.cast(
-                tf.math.count_nonzero(y_pred * y_true, axis=self.axis),
-                self.dtype))
-        # false positive
-        self.false_positives.assign_add(
-            tf.cast(
-                tf.math.count_nonzero(y_pred * (y_true - 1), axis=self.axis),
-                self.dtype))
-        # false negative
-        self.false_negatives.assign_add(
-            tf.cast(
-                tf.math.count_nonzero((y_pred - 1) * y_true, axis=self.axis),
-                self.dtype))
-        if self.average == 'weighted':
-            # variable to hold intermediate weights
-            self.weights_intermediate.assign_add(
-                tf.cast(tf.reduce_sum(y_true, axis=self.axis), self.dtype))
+        def _count_non_zero(val):
+            non_zeros = tf.math.count_nonzero(val, axis=self.axis)
+            return tf.cast(non_zeros, self.dtype)
+
+        self.true_positives.assign_add(_count_non_zero(y_pred * y_true))
+        self.false_positives.assign_add(_count_non_zero(y_pred * (y_true - 1)))
+        self.false_negatives.assign_add(_count_non_zero((y_pred - 1) * y_true))
+        self.weights_intermediate.assign_add(_count_non_zero(y_true))
 
     def result(self):
-        p_sum = tf.cast(self.true_positives + self.false_positives, self.dtype)
-        # calculate precision
-        precision = tf.math.divide_no_nan(self.true_positives, p_sum)
-
-        r_sum = tf.cast(self.true_positives + self.false_negatives, self.dtype)
-        # calculate recall
-        recall = tf.math.divide_no_nan(self.true_positives, r_sum)
-        # intermediate calculations
+        precision = tf.math.divide_no_nan(
+            self.true_positives, self.true_positives + self.false_positives)
+        recall = tf.math.divide_no_nan(
+            self.true_positives, self.true_positives + self.false_negatives)
+
         mul_value = precision * recall
         add_value = (tf.math.square(self.beta) * precision) + recall
-        f1_int = (1 + tf.math.square(self.beta)) * (tf.math.divide_no_nan(
-            mul_value, add_value))
-        # f1 score
-        if self.average is not None:
-            f1_score = tf.reduce_mean(f1_int)
-        else:
-            f1_score = f1_int
-        # condition for weighted f1 score
+        mean = (tf.math.divide_no_nan(mul_value, add_value))
+        f1_score = mean * (1 + tf.math.square(self.beta))
+
         if self.average == 'weighted':
-            f1_int_weights = tf.math.divide_no_nan(
+            weights = tf.math.divide_no_nan(
                 self.weights_intermediate,
                 tf.reduce_sum(self.weights_intermediate))
-            # weighted f1 score calculation
-            f1_score = tf.reduce_sum(f1_int * f1_int_weights)
+            f1_score = tf.reduce_sum(f1_score * weights)
+
+        elif self.average is not None:  # [micro, macro]
+            f1_score = tf.reduce_mean(f1_score)
 
         return f1_score
 
@@ -237,111 +172,70 @@ def get_config(self):
             "average": self.average,
             "beta": self.beta,
         }
+
+        if self.threshold is not None:
+            config["threshold"] = self.threshold
+
         base_config = super(FBetaScore, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
 
     def reset_states(self):
-        # reset state of the variables to zero
-        if self.average == 'micro':
-            self.true_positives.assign(0)
-            self.false_positives.assign(0)
-            self.false_negatives.assign(0)
-        else:
-            self.true_positives.assign(np.zeros(self.num_classes), np.float32)
-            self.false_positives.assign(np.zeros(self.num_classes), np.float32)
-            self.false_negatives.assign(np.zeros(self.num_classes), np.float32)
-            self.weights_intermediate.assign(
-                np.zeros(self.num_classes), np.float32)
+        self.true_positives.assign(tf.zeros(self.init_shape, self.dtype))
+        self.false_positives.assign(tf.zeros(self.init_shape, self.dtype))
+        self.false_negatives.assign(tf.zeros(self.init_shape, self.dtype))
+        self.weights_intermediate.assign(tf.zeros(self.init_shape, self.dtype))
 
 
+@keras_utils.register_keras_custom_object
 class F1Score(FBetaScore):
-    """Computes F1 micro, macro or weighted based on the user's choice.
+    """Computes F-1 Score.
 
-    F1 score is the weighted average of precision and
-    recall. Output range is [0, 1]. This works for both
-    multi-class and multi-label classification.
+    It is the harmonic mean of precision and recall.
+    Output range is [0, 1]. Works for both multi-class
+    and multi-label classification.
 
-    F-1 = (2) * ((precision * recall) / (precision + recall))
+    F-1 = 2 * (precision * recall) / (precision + recall)
 
     Args:
-       num_classes : Number of unique classes in the dataset.
-       average : Type of averaging to be performed on data.
-                 Acceptable values are `None`, `micro`, `macro` and
-                 `weighted`.
-                 Default value is None.
-       beta : float
-              Determines the weight of precision and recall in harmonic
-              mean. Its value is 1.0 for F1 score.
+        num_classes: Number of unique classes in the dataset.
+        average: Type of averaging to be performed on data.
+            Acceptable values are `None`, `micro`, `macro`
+            and `weighted`. Default value is None.
+        threshold: Elements of `y_pred` above threshold are
+            considered to be 1, and the rest 0. If threshold is
+            None, the argmax is converted to 1, and the rest 0.
 
     Returns:
-       F1 Score: float
+        F-1 Score: float
 
     Raises:
-       ValueError: If the `average` has values other than
-       [None, micro, macro, weighted].
-
-       ValueError: If the `beta` value is less than or equal
-       to 0.
+        ValueError: If the `average` has values other than
+        [None, micro, macro, weighted].
 
     `average` parameter behavior:
+        None: Scores for each class are returned
+
+        micro: True positivies, false positives and
+            false negatives are computed globally.
 
-    1. If `None` is specified as an input, scores for each
-       class are returned.
-
-    2. If `micro` is specified, metrics like true positivies,
-       false positives and false negatives are computed
-       globally.
-
-    3. If `macro` is specified, metrics like true positivies,
-       false positives and false negatives are computed for
-       each class and their unweighted mean is returned.
-       Imbalance in dataset is not taken into account for
-       calculating the score
-
-    4. If `weighted` is specified, metrics are computed for
-       each class and returns the mean weighted by the
-       number of true instances in each class taking data
-       imbalance into account.
-
-    Usage:
-    ```python
-    actuals = tf.constant([[1, 1, 0],[1, 0, 0]],
-              dtype=tf.int32)
-    preds = tf.constant([[1, 0, 0],[1, 0, 1]],
-              dtype=tf.int32)
-    # F1 Micro
-    output = tfa.metrics.F1Score(num_classes=3,
-              average='micro')
-    output.update_state(actuals, preds)
-    print('F1 Micro score is: ',
-            output.result().numpy()) # 0.6666667
-    # F1 Macro
-    output = tfa.metrics.F1Score(num_classes=3,
-                average='macro')
-    output.update_state(actuals, preds)
-    print('F1 Macro score is: ',
-            output.result().numpy()) # 0.33333334
-    # F1 weighted
-    output = tfa.metrics.F1Score(num_classes=3,
-              average='weighted')
-    output.update_state(actuals, preds)
-    print('F1 Weighted score is: ',
-            output.result().numpy()) # 0.6666667
-    # F1 score for each class (average=None).
-    output = tfa.metrics.F1Score(num_classes=3)
-    output.update_state(actuals, preds)
-    print('F1 score is: ',
-            output.result().numpy()) # [1. 0. 0.]
-    ```
+        macro: True positivies, false positives and
+            false negatives are computed for each class
+            and their unweighted mean is returned.
+
+        weighted: Metrics are computed for each class
+            and returns the mean weighted by the
+            number of true instances in each class.
     """
 
-    def __init__(self, num_classes, average, name='f1_score',
+    def __init__(self,
+                 num_classes,
+                 average=None,
+                 threshold=None,
+                 name='f1_score',
                  dtype=tf.float32):
         super(F1Score, self).__init__(
-            num_classes, average, 1.0, name=name, dtype=dtype)
+            num_classes, average, 1.0, threshold, name=name, dtype=dtype)
 
-    # TODO: Add sample_weight support, currently it is
-    # ignored during calculations.
     def get_config(self):
         base_config = super(F1Score, self).get_config()
         del base_config["beta"]
diff --git a/tensorflow_addons/metrics/utils.py b/tensorflow_addons/metrics/utils.py
--- a/tensorflow_addons/metrics/utils.py
+++ b/tensorflow_addons/metrics/utils.py
@@ -18,6 +18,7 @@
 from __future__ import print_function
 
 import six
+import numpy as np
 import tensorflow as tf
 
 
@@ -65,3 +66,18 @@ def get_config(self):
             config[k] = v
         base_config = super(MeanMetricWrapper, self).get_config()
         return dict(list(base_config.items()) + list(config.items()))
+
+
+def _get_model(metric, num_output):
+    # Test API comptibility with tf.keras Model
+    model = tf.keras.Sequential()
+    model.add(tf.keras.layers.Dense(64, activation='relu'))
+    model.add(tf.keras.layers.Dense(num_output, activation='softmax'))
+    model.compile(
+        optimizer='adam',
+        loss='categorical_crossentropy',
+        metrics=['acc', metric])
+
+    data = np.random.random((10, 3))
+    labels = np.random.random((10, num_output))
+    model.fit(data, labels, epochs=1, batch_size=5, verbose=0)


Test patch:
diff --git a/tensorflow_addons/metrics/f1_test.py b/tensorflow_addons/metrics/f1_test.py
deleted file mode 100755
--- a/tensorflow_addons/metrics/f1_test.py
+++ /dev/null
@@ -1,130 +0,0 @@
-# Copyright 2019 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Tests F1 metrics."""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-import tensorflow as tf
-from tensorflow_addons.metrics import F1Score
-from tensorflow_addons.utils import test_utils
-from tensorflow.keras import layers
-import numpy as np
-
-
-@test_utils.run_all_in_graph_and_eager_modes
-class F1ScoreTest(tf.test.TestCase):
-    def test_config(self):
-        f1_obj = F1Score(num_classes=3, average=None)
-        self.assertEqual(f1_obj.name, 'f1_score')
-        self.assertEqual(f1_obj.dtype, tf.float32)
-        self.assertEqual(f1_obj.num_classes, 3)
-        self.assertEqual(f1_obj.average, None)
-        # Check save and restore config
-        f1_obj2 = F1Score.from_config(f1_obj.get_config())
-        self.assertEqual(f1_obj2.name, 'f1_score')
-        self.assertEqual(f1_obj2.dtype, tf.float32)
-        self.assertEqual(f1_obj2.num_classes, 3)
-        self.assertEqual(f1_obj2.average, None)
-
-    def initialize_vars(self, average):
-        # initialize variables
-        f1_obj = F1Score(num_classes=3, average=average)
-
-        self.evaluate(tf.compat.v1.variables_initializer(f1_obj.variables))
-
-        return f1_obj
-
-    def update_obj_states(self, f1_obj, actuals, preds):
-        # update state variable values
-        update_op = f1_obj.update_state(actuals, preds)
-        self.evaluate(update_op)
-
-    def check_results(self, f1_obj, value):
-        # check result
-        self.assertAllClose(value, self.evaluate(f1_obj.result()), atol=1e-5)
-
-    def _test_f1(self, avg, act, pred, res):
-        f1_init = self.initialize_vars(avg)
-        self.update_obj_states(f1_init, act, pred)
-        self.check_results(f1_init, res)
-
-    def _test_f1_score(self, actuals, preds, res):
-        # test for three average values with beta value as 1.0
-        for avg in ['micro', 'macro', 'weighted']:
-            self._test_f1(avg, actuals, preds, res)
-
-    # test for perfect f1 score
-    def test_f1_perfect_score(self):
-        actuals = tf.constant([[1, 1, 1], [1, 0, 0], [1, 1, 0]],
-                              dtype=tf.int32)
-        preds = tf.constant([[1, 1, 1], [1, 0, 0], [1, 1, 0]], dtype=tf.int32)
-        self._test_f1_score(actuals, preds, 1.0)
-
-    # test for worst f1 score
-    def test_f1_worst_score(self):
-        actuals = tf.constant([[1, 1, 1], [1, 0, 0], [1, 1, 0]],
-                              dtype=tf.int32)
-        preds = tf.constant([[0, 0, 0], [0, 1, 0], [0, 0, 1]], dtype=tf.int32)
-        self._test_f1_score(actuals, preds, 0.0)
-
-    # test for random f1 score
-    def test_f1_random_score(self):
-        actuals = tf.constant([[1, 1, 1], [1, 0, 0], [1, 1, 0]],
-                              dtype=tf.int32)
-        preds = tf.constant([[0, 0, 1], [1, 1, 0], [1, 1, 1]], dtype=tf.int32)
-        # Use absl parameterized test here if possible
-        test_params = [['micro', 0.6666667], ['macro', 0.65555555],
-                       ['weighted', 0.67777777]]
-
-        for avg, res in test_params:
-            self._test_f1(avg, actuals, preds, res)
-
-    # test for random f1 score with average as None
-    def test_f1_random_score_none(self):
-        actuals = tf.constant(
-            [[1, 0, 0], [0, 1, 0], [0, 0, 1], [1, 0, 0], [0, 1, 0], [0, 0, 1]],
-            dtype=tf.int32)
-        preds = tf.constant(
-            [[1, 0, 0], [0, 1, 0], [0, 0, 1], [1, 0, 0], [1, 0, 0], [0, 0, 1]],
-            dtype=tf.int32)
-
-        # Use absl parameterized test here if possible
-        test_params = [[None, [0.8, 0.6666667, 1.]]]
-
-        for avg, res in test_params:
-            self._test_f1(avg, actuals, preds, res)
-
-    # Keras model check
-    def test_keras_model(self):
-        model = tf.keras.Sequential()
-        model.add(layers.Dense(64, activation='relu'))
-        model.add(layers.Dense(64, activation='relu'))
-        model.add(layers.Dense(1, activation='softmax'))
-        fb = F1Score(1, 'macro')
-        model.compile(
-            optimizer='rmsprop',
-            loss='categorical_crossentropy',
-            metrics=['acc', fb])
-        # data preparation
-        data = np.random.random((10, 3))
-        labels = np.random.random((10, 1))
-        labels = np.where(labels > 0.5, 1, 0)
-        model.fit(data, labels, epochs=1, batch_size=32, verbose=0)
-
-
-if __name__ == '__main__':
-    tf.test.main()
diff --git a/tensorflow_addons/metrics/f_test.py b/tensorflow_addons/metrics/f_test.py
new file mode 100644
--- /dev/null
+++ b/tensorflow_addons/metrics/f_test.py
@@ -0,0 +1,149 @@
+# Copyright 2019 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Tests F beta metrics."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+from absl.testing import parameterized
+
+import tensorflow as tf
+from tensorflow_addons.metrics import FBetaScore, F1Score, utils
+from tensorflow_addons.utils import test_utils
+
+
+@test_utils.run_all_in_graph_and_eager_modes
+class FBetaScoreTest(tf.test.TestCase, parameterized.TestCase):
+    def test_config(self):
+        fbeta_obj = FBetaScore(
+            num_classes=3, beta=0.5, threshold=0.3, average=None)
+        self.assertEqual(fbeta_obj.beta, 0.5)
+        self.assertEqual(fbeta_obj.average, None)
+        self.assertEqual(fbeta_obj.threshold, 0.3)
+        self.assertEqual(fbeta_obj.num_classes, 3)
+        self.assertEqual(fbeta_obj.dtype, tf.float32)
+
+        # Check save and restore config
+        fbeta_obj2 = FBetaScore.from_config(fbeta_obj.get_config())
+        self.assertEqual(fbeta_obj2.beta, 0.5)
+        self.assertEqual(fbeta_obj2.average, None)
+        self.assertEqual(fbeta_obj2.threshold, 0.3)
+        self.assertEqual(fbeta_obj2.num_classes, 3)
+        self.assertEqual(fbeta_obj2.dtype, tf.float32)
+
+    def _test_tf(self, avg, beta, act, pred, threshold):
+        act = tf.constant(act, tf.float32)
+        pred = tf.constant(pred, tf.float32)
+
+        fbeta = FBetaScore(3, avg, beta, threshold)
+        self.evaluate(tf.compat.v1.variables_initializer(fbeta.variables))
+        self.evaluate(fbeta.update_state(act, pred))
+        return self.evaluate(fbeta.result())
+
+    def _test_fbeta_score(self, actuals, preds, avg, beta_val, result,
+                          threshold):
+        tf_score = self._test_tf(avg, beta_val, actuals, preds, threshold)
+        self.assertAllClose(tf_score, result, atol=1e-7)
+
+    def test_fbeta_perfect_score(self):
+        preds = [[0.7, 0.7, 0.7], [1, 0, 0], [0.9, 0.8, 0]]
+        actuals = [[1, 1, 1], [1, 0, 0], [1, 1, 0]]
+
+        for avg_val in ['micro', 'macro', 'weighted']:
+            for beta in [0.5, 1.0, 2.0]:
+                self._test_fbeta_score(actuals, preds, avg_val, beta, 1.0,
+                                       0.66)
+
+    def test_fbeta_worst_score(self):
+        preds = [[0.7, 0.7, 0.7], [1, 0, 0], [0.9, 0.8, 0]]
+        actuals = [[0, 0, 0], [0, 1, 0], [0, 0, 1]]
+
+        for avg_val in ['micro', 'macro', 'weighted']:
+            for beta in [0.5, 1.0, 2.0]:
+                self._test_fbeta_score(actuals, preds, avg_val, beta, 0.0,
+                                       0.66)
+
+    @parameterized.parameters([[None, 0.5, [0.71428573, 0.5, 0.833334]],
+                               [None, 1.0, [0.8, 0.5, 0.6666667]],
+                               [None, 2.0, [0.9090904, 0.5, 0.555556]],
+                               ['micro', 0.5, 0.6666667],
+                               ['micro', 1.0, 0.6666667],
+                               ['micro', 2.0, 0.6666667],
+                               ['macro', 0.5, 0.6825397],
+                               ['macro', 1.0, 0.6555555],
+                               ['macro', 2.0, 0.6548822],
+                               ['weighted', 0.5, 0.6825397],
+                               ['weighted', 1.0, 0.6555555],
+                               ['weighted', 2.0, 0.6548822]])
+    def test_fbeta_random_score(self, avg_val, beta, result):
+        preds = [[0.7, 0.7, 0.7], [1, 0, 0], [0.9, 0.8, 0]]
+        actuals = [[0, 0, 1], [1, 1, 0], [1, 1, 1]]
+        self._test_fbeta_score(actuals, preds, avg_val, beta, result, 0.66)
+
+    @parameterized.parameters([[None, 0.5, [0.9090904, 0.555556, 1.0]],
+                               [None, 1.0, [0.8, 0.6666667, 1.0]],
+                               [None, 2.0, [0.71428573, 0.833334, 1.0]],
+                               ['micro', 0.5, 0.833334],
+                               ['micro', 1.0, 0.833334],
+                               ['micro', 2.0, 0.833334],
+                               ['macro', 0.5, 0.821549],
+                               ['macro', 1.0, 0.822222],
+                               ['macro', 2.0, 0.849206],
+                               ['weighted', 0.5, 0.880471],
+                               ['weighted', 1.0, 0.844445],
+                               ['weighted', 2.0, 0.829365]])
+    def test_fbeta_random_score_none(self, avg_val, beta, result):
+        preds = [[0.9, 0.1, 0], [0.2, 0.6, 0.2], [0, 0, 1], [0.4, 0.3, 0.3],
+                 [0, 0.9, 0.1], [0, 0, 1]]
+        actuals = [[1, 0, 0], [0, 1, 0], [0, 0, 1], [1, 0, 0], [1, 0, 0],
+                   [0, 0, 1]]
+        self._test_fbeta_score(actuals, preds, avg_val, beta, result, None)
+
+    def test_keras_model(self):
+        fbeta = FBetaScore(5, 'micro', 1.0)
+        utils._get_model(fbeta, 5)
+
+
+@test_utils.run_all_in_graph_and_eager_modes
+class F1ScoreTest(tf.test.TestCase):
+    def test_eq(self):
+        f1 = F1Score(3)
+        fbeta = FBetaScore(3, beta=1.0)
+        self.evaluate(tf.compat.v1.variables_initializer(f1.variables))
+        self.evaluate(tf.compat.v1.variables_initializer(fbeta.variables))
+
+        preds = [[0.9, 0.1, 0], [0.2, 0.6, 0.2], [0, 0, 1], [0.4, 0.3, 0.3],
+                 [0, 0.9, 0.1], [0, 0, 1]]
+        actuals = [[1, 0, 0], [0, 1, 0], [0, 0, 1], [1, 0, 0], [1, 0, 0],
+                   [0, 0, 1]]
+
+        self.evaluate(fbeta.update_state(actuals, preds))
+        self.evaluate(f1.update_state(actuals, preds))
+        self.assertAllClose(
+            self.evaluate(fbeta.result()), self.evaluate(f1.result()))
+
+    def test_keras_model(self):
+        f1 = F1Score(5)
+        utils._get_model(f1, 5)
+
+    def test_config(self):
+        f1 = F1Score(3)
+        config = f1.get_config()
+        self.assertFalse("beta" in config)
+
+
+if __name__ == '__main__':
+    tf.test.main()
diff --git a/tensorflow_addons/metrics/fbeta_test.py b/tensorflow_addons/metrics/fbeta_test.py
deleted file mode 100644
--- a/tensorflow_addons/metrics/fbeta_test.py
+++ /dev/null
@@ -1,136 +0,0 @@
-# Copyright 2019 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Tests F beta metrics."""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-import tensorflow as tf
-from tensorflow_addons.metrics import FBetaScore
-from tensorflow_addons.utils import test_utils
-from tensorflow.keras import layers
-import numpy as np
-
-
-@test_utils.run_all_in_graph_and_eager_modes
-class FBetaScoreTest(tf.test.TestCase):
-    def test_config(self):
-        fbeta_obj = FBetaScore(num_classes=3, beta=0.5, average=None)
-        self.assertEqual(fbeta_obj.beta, 0.5)
-        self.assertEqual(fbeta_obj.average, None)
-        self.assertEqual(fbeta_obj.num_classes, 3)
-        self.assertEqual(fbeta_obj.dtype, tf.float32)
-        # Check save and restore config
-        fbeta_obj2 = FBetaScore.from_config(fbeta_obj.get_config())
-        self.assertEqual(fbeta_obj2.beta, 0.5)
-        self.assertEqual(fbeta_obj2.average, None)
-        self.assertEqual(fbeta_obj2.num_classes, 3)
-        self.assertEqual(fbeta_obj2.dtype, tf.float32)
-
-    def initialize_vars(self, beta_val, average):
-        # initialize variables
-        fbeta_obj = FBetaScore(num_classes=3, beta=beta_val, average=average)
-
-        self.evaluate(tf.compat.v1.variables_initializer(fbeta_obj.variables))
-
-        return fbeta_obj
-
-    def update_obj_states(self, fbeta_obj, actuals, preds):
-        # update state variables values
-        update_op = fbeta_obj.update_state(actuals, preds)
-        self.evaluate(update_op)
-
-    def check_results(self, fbeta_obj, value):
-        # check results
-        self.assertAllClose(
-            value, self.evaluate(fbeta_obj.result()), atol=1e-5)
-
-    def _test_fbeta(self, avg, beta, act, pred, res):
-        fbeta = self.initialize_vars(beta, avg)
-        self.update_obj_states(fbeta, act, pred)
-        self.check_results(fbeta, res)
-
-    def _test_fbeta_score(self, actuals, preds, res):
-        # This function tests for three average values and
-        # two beta values
-        for avg in ['micro', 'macro', 'weighted']:
-            for beta_val in [0.5, 2.0]:
-                self._test_fbeta(avg, beta_val, actuals, preds, res)
-
-    # test for the perfect score
-    def test_fbeta_perfect_score(self):
-        actuals = tf.constant([[1, 1, 1], [1, 0, 0], [1, 1, 0]],
-                              dtype=tf.int32)
-        preds = tf.constant([[1, 1, 1], [1, 0, 0], [1, 1, 0]], dtype=tf.int32)
-        self._test_fbeta_score(actuals, preds, 1.0)
-
-    # test for the worst score
-    def test_fbeta_worst_score(self):
-        actuals = tf.constant([[1, 1, 1], [1, 0, 0], [1, 1, 0]],
-                              dtype=tf.int32)
-        preds = tf.constant([[0, 0, 0], [0, 1, 0], [0, 0, 1]], dtype=tf.int32)
-        self._test_fbeta_score(actuals, preds, 0.0)
-
-    # test for the random score
-    def test_fbeta_random_score(self):
-        actuals = tf.constant([[1, 1, 1], [1, 0, 0], [1, 1, 0]],
-                              dtype=tf.int32)
-        preds = tf.constant([[0, 0, 1], [1, 1, 0], [1, 1, 1]], dtype=tf.int32)
-
-        # test parameters
-        test_params = [['micro', 0.5, 0.666667], ['macro', 0.5, 0.654882],
-                       ['weighted', 0.5, 0.71380], ['micro', 2.0, 0.666667],
-                       ['macro', 2.0, 0.68253], ['weighted', 2.0, 0.66269]]
-
-        for avg, beta, res in test_params:
-            self._test_fbeta(avg, beta, actuals, preds, res)
-
-    # Test for the random score with average value as None
-    def test_fbeta_random_score_none(self):
-        actuals = tf.constant(
-            [[1, 0, 0], [0, 1, 0], [0, 0, 1], [1, 0, 0], [0, 1, 0], [0, 0, 1]],
-            dtype=tf.int32)
-        preds = tf.constant(
-            [[1, 0, 0], [0, 1, 0], [0, 0, 1], [1, 0, 0], [1, 0, 0], [0, 0, 1]],
-            dtype=tf.int32)
-
-        # test parameters
-        test_params = [[0.5, [0.71428573, 0.8333334, 1.]],
-                       [2.0, [0.90909094, 0.5555556, 1.]]]
-
-        for beta, res in test_params:
-            self._test_fbeta(None, beta, actuals, preds, res)
-
-    # Keras model check
-    def test_keras_model(self):
-        model = tf.keras.Sequential()
-        model.add(layers.Dense(64, activation='relu'))
-        model.add(layers.Dense(64, activation='relu'))
-        model.add(layers.Dense(1, activation='softmax'))
-        fb = FBetaScore(1, 'macro')
-        model.compile(
-            optimizer='rmsprop',
-            loss='categorical_crossentropy',
-            metrics=['acc', fb])
-        # data preparation
-        data = np.random.random((10, 3))
-        labels = np.random.random((10, 1))
-        labels = np.where(labels > 0.5, 1, 0)
-        model.fit(data, labels, epochs=1, batch_size=32, verbose=0)
-
-
-if __name__ == '__main__':
-    tf.test.main()