Repository: tensorflow/addons

Issue:
Graduate LayerNormalization layer to TF core.
**System information**
- TensorFlow version (you are using):
- TensorFlow Addons version:
- Is it in the tf.contrib (if so, where):
- Are you willing to contribute it (yes/no):
- Are you willing to maintain it going forward? (yes/no):

**Describe the feature and the current behavior/state.**
LayerBatchNormalization has been moved to core here: https://github.com/tensorflow/tensorflow/blob/43a255d44035f2a68585288876e2ccbb77cb70de/tensorflow/python/keras/layers/normalization.py#L807

Can we graduate it from Addons?

**Will this change the current api? How?**
Yes, this will delete the LayerBatchNormalization API in Addons.

**Who will benefit with this feature?**

**Any Other info.**



Discussion:
Per the discussions in https://github.com/tensorflow/addons/pull/14 . We're going to request that LayerNormalization be removed from core. Although now it looks like the experimental tag has been removed from core? cc @karmel as this was not the roadmap we discussed.

If it does stay in core, I'd recommend that we use a more general implemenetation of LayernNorm that includes GroupNorm and InstanceNorm as we have done in addons.
@seanpmorgan After some discussion, it seemed best to graduate the implementation from addons to core, as we were seeing increased requests and usage. The Addons implementations of Group and InstanceNorm would then remain, and could build off the core implementation of LayerNorm. We figured this would be a win for all, as it is the first example of moving from Addons to core, and the more specialized implementations (Instance + GroupNorm) still have a proper home here. Does that work for you?
@karmel thanks for the follow up... yes that works for us. My only hesitation would be the comparisons in the GN paper that show LN to be a bit less flexible than GN and worse performing. Perhaps LN is being used because of legacy examples... but that's not really our battle to fight.

> Relation to Layer Normalization [3]. GN becomes LN if we set the group number as G = 1. LN assumes all channels in a layer make “similar contributions” [3]. Unlike the case of fully-connected layers studied in [3], this assumption can be less valid with the presence of convolutions, as discussed
in [3]. GN is less restricted than LN, because each group of channels (instead of all of them) are assumed to subject to the shared mean and variance; the model still has flexibility of learning a different distribution for each group. This leads to improved representational power of GN over LN,
as shown by the lower training and validation error in experiments (Figure 4).

![image](https://user-images.githubusercontent.com/18154355/56151212-c52ba280-5f7e-11e9-9ba7-1f50b98ce16e.png)
https://arxiv.org/abs/1803.08494


@Smokrow Would you mind submitting a PR to remove LayerNormalization so we can keep our apis in sync with TF core. Thanks.
@seanpmorgan Thanks! One noticeable difference between core LN and Addons LN is the default value of epsilon. TF core uses 1e-3 to keep it consistent with that of Keras BN; While I see Addons implementation uses 1e-5.
@seanpmorgan I will open a PR tomorrow morning.

@karmel if I can help out with anything during the move please let me know. 
@karmel Layer Normalization is a more specialized case of group normalization. 
I can just remove this special case but this would of course kinda break the inheritance. Maybe it would make more sense to move all 3 layers in one go since the specialization is just 7 lines of code. 

For example the current implementation of LayerNorm looks like this:

``` python
@keras_utils.register_keras_custom_object
class LayerNormalization(GroupNormalization):
    def __init__(self, **kwargs):
        if "groups" in kwargs:
            logging.warning("The given value for groups will be overwritten.")
        kwargs["groups"] = 1
        super(LayerNormalization, self).__init__(**kwargs)
``` 

@Smokrow I think the most pressing thing to do is to align our api's and remove LayerNormalization. Also @yhliang2018 suggestion to use 1e-3 is better implemented sooner rather than later before we acquire too many users. 

I do agree with you, but I imagine the process of adding two new layers will take a while and I'd prefer not to have duplicate functionality now that LayerNormalization is not experimental.

Resolving patch:
diff --git a/tensorflow_addons/layers/__init__.py b/tensorflow_addons/layers/__init__.py
--- a/tensorflow_addons/layers/__init__.py
+++ b/tensorflow_addons/layers/__init__.py
@@ -21,7 +21,6 @@
 from tensorflow_addons.layers.maxout import Maxout
 from tensorflow_addons.layers.normalizations import GroupNormalization
 from tensorflow_addons.layers.normalizations import InstanceNormalization
-from tensorflow_addons.layers.normalizations import LayerNormalization
 from tensorflow_addons.layers.poincare import PoincareNormalize
 from tensorflow_addons.layers.sparsemax import Sparsemax
 from tensorflow_addons.layers.wrappers import WeightNormalization
diff --git a/tensorflow_addons/layers/normalizations.py b/tensorflow_addons/layers/normalizations.py
--- a/tensorflow_addons/layers/normalizations.py
+++ b/tensorflow_addons/layers/normalizations.py
@@ -74,7 +74,7 @@ class GroupNormalization(tf.keras.layers.Layer):
     def __init__(self,
                  groups=2,
                  axis=-1,
-                 epsilon=1e-5,
+                 epsilon=1e-3,
                  center=True,
                  scale=True,
                  beta_initializer='zeros',
@@ -273,49 +273,6 @@ def _create_broadcast_shape(self, input_shape):
         return broadcast_shape
 
 
-@keras_utils.register_keras_custom_object
-class LayerNormalization(GroupNormalization):
-    """Layer normalization layer.
-
-    Layer Normalization is an specific case of ```GroupNormalization```since it
-    normalizes all features of a layer. The Groupsize is 1.
-    Empirically, its accuracy is more stable than batch norm in a wide
-    range of small batch sizes, if learning rate is adjusted linearly
-    with batch sizes.
-
-    Arguments
-        axis: Integer, the axis that should be normalized.
-        epsilon: Small float added to variance to avoid dividing by zero.
-        center: If True, add offset of `beta` to normalized tensor.
-            If False, `beta` is ignored.
-        scale: If True, multiply by `gamma`.
-            If False, `gamma` is not used.
-        beta_initializer: Initializer for the beta weight.
-        gamma_initializer: Initializer for the gamma weight.
-        beta_regularizer: Optional regularizer for the beta weight.
-        gamma_regularizer: Optional regularizer for the gamma weight.
-        beta_constraint: Optional constraint for the beta weight.
-        gamma_constraint: Optional constraint for the gamma weight.
-
-    Input shape
-        Arbitrary. Use the keyword argument `input_shape`
-        (tuple of integers, does not include the samples axis)
-        when using this layer as the first layer in a model.
-
-    Output shape
-        Same shape as input.
-
-    References
-        - [Layer Normalization](https://arxiv.org/abs/1607.06450)
-    """
-
-    def __init__(self, **kwargs):
-        if "groups" in kwargs:
-            logging.warning("The given value for groups will be overwritten.")
-        kwargs["groups"] = 1
-        super(LayerNormalization, self).__init__(**kwargs)
-
-
 @keras_utils.register_keras_custom_object
 class InstanceNormalization(GroupNormalization):
     """Instance normalization layer.


Test patch:
diff --git a/tensorflow_addons/layers/normalizations_test.py b/tensorflow_addons/layers/normalizations_test.py
--- a/tensorflow_addons/layers/normalizations_test.py
+++ b/tensorflow_addons/layers/normalizations_test.py
@@ -21,28 +21,22 @@
 
 from tensorflow_addons.layers.normalizations import GroupNormalization
 from tensorflow_addons.layers.normalizations import InstanceNormalization
-from tensorflow_addons.layers.normalizations import LayerNormalization
 from tensorflow_addons.utils import test_utils
 
 
 class NormalizationTest(tf.test.TestCase):
 
     # ------------Tests to ensure proper inheritance. If these suceed you can
-    # test for Instance norm and Layernorm by setting Groupnorm groups = -1 or 1
+    # test for Instance norm by setting Groupnorm groups = -1
     def test_inheritance(self):
-        self.assertTrue(issubclass(LayerNormalization, GroupNormalization))
         self.assertTrue(issubclass(InstanceNormalization, GroupNormalization))
-        self.assertTrue(LayerNormalization.build == GroupNormalization.build)
         self.assertTrue(
             InstanceNormalization.build == GroupNormalization.build)
-        self.assertTrue(LayerNormalization.call == GroupNormalization.call)
         self.assertTrue(InstanceNormalization.call == GroupNormalization.call)
 
     def test_groups_after_init(self):
         layers = InstanceNormalization()
         self.assertTrue(layers.groups == -1)
-        layers = LayerNormalization()
-        self.assertTrue(layers.groups == 1)
 
     # ------------------------------------------------------------------------------
 
@@ -167,11 +161,6 @@ def test_weights(self):
         self.assertEqual(len(layer.trainable_weights), 0)
         self.assertEqual(len(layer.weights), 0)
 
-        layer = LayerNormalization()
-        layer.build((None, 3, 4))
-        self.assertEqual(len(layer.trainable_weights), 2)
-        self.assertEqual(len(layer.weights), 2)
-
         layer = InstanceNormalization()
         layer.build((None, 3, 4))
         self.assertEqual(len(layer.trainable_weights), 2)
@@ -208,15 +197,6 @@ def test_groupnorm_flat(self):
             self.assertTrue(hasattr(model.layers[0], 'gamma'))
             self.assertTrue(hasattr(model.layers[0], 'beta'))
 
-    @test_utils.run_in_graph_and_eager_modes
-    def test_layernorm_flat(self):
-        # Check basic usage of layernorm
-
-        model = self._create_and_fit_Sequential_model(LayerNormalization(),
-                                                      (64,))
-        self.assertTrue(hasattr(model.layers[0], 'gamma'))
-        self.assertTrue(hasattr(model.layers[0], 'beta'))
-
     @test_utils.run_in_graph_and_eager_modes
     def test_instancenorm_flat(self):
         # Check basic usage of instancenorm