internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: tensorflow/addons

Issue:
Proposal: Upload the nightlies at each commit on the master branch
We already build the wheels already anyway. It's just missing the push to pypi. Yes/No?


Discussion:
Good question. It would only require a minor change to our nightly versioning to add timestamps:
https://github.com/tensorflow/addons/blob/master/setup.py#L55

Seeing as our package is only ~1MB in size I'm not overly concerned with the space it takes up on pypi. It is something to think about though if down the line we find a need to increase package size substantially.

Given that info, I think the benefits outweigh then downsides and I would be in favor of adding this.

Next to tensorflow and their 500MB uploaded to PyPI every day, we have some margin :laughing: 

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -52,7 +52,7 @@
     exec(fp.read(), version)
 
 if project_name == TFA_NIGHTLY:
-    version["__version__"] += datetime.strftime(datetime.today(), "%Y%m%d")
+    version["__version__"] += datetime.now().strftime("%Y%m%d%H%M%S")
 
 with open("requirements.txt") as f:
     required_pkgs = f.read().splitlines()


Test patch:
Repository: tensorflow/addons

Issue:
Dropping support for tensorflow < 2.2.
We have three pull requests that need TF 2.2 and above: #1733 #1726 #905 

They are blocked until we drop support, since Addons 0.9 was released with support for both tf 2.1 and 2.2, I don't think it's going to be a very big inconvenience for users if we drop support for 2.1 now. They have a good fallback.


Discussion:
Agree. Going to mark those PRs as blocked until we swap our CI to only test 2.2

Resolving patch:
diff --git a/tensorflow_addons/utils/ensure_tf_install.py b/tensorflow_addons/utils/ensure_tf_install.py
--- a/tensorflow_addons/utils/ensure_tf_install.py
+++ b/tensorflow_addons/utils/ensure_tf_install.py
@@ -23,7 +23,7 @@
 
 import tensorflow as tf
 
-MIN_TF_VERSION = "2.1.0"
+MIN_TF_VERSION = "2.2.0"
 MAX_TF_VERSION = "2.3.0"
 
 
diff --git a/tensorflow_addons/utils/resource_loader.py b/tensorflow_addons/utils/resource_loader.py
--- a/tensorflow_addons/utils/resource_loader.py
+++ b/tensorflow_addons/utils/resource_loader.py
@@ -20,8 +20,8 @@
 
 import tensorflow as tf
 
-MIN_TF_VERSION_FOR_ABI_COMPATIBILITY = "2.1.0"
-MAX_TF_VERSION_FOR_ABI_COMPATIBILITY = "2.2.0"
+MIN_TF_VERSION_FOR_ABI_COMPATIBILITY = "2.2.0"
+MAX_TF_VERSION_FOR_ABI_COMPATIBILITY = "2.3.0"
 abi_warning_already_raised = False
 SKIP_CUSTOM_OPS = False
 


Test patch:
diff --git a/tools/docker/cpu_tests.Dockerfile b/tools/docker/cpu_tests.Dockerfile
--- a/tools/docker/cpu_tests.Dockerfile
+++ b/tools/docker/cpu_tests.Dockerfile
@@ -1,7 +1,7 @@
 #syntax=docker/dockerfile:1.1.5-experimental
 FROM python:3.5 as build_wheel
 
-ARG TF_VERSION=2.1.0
+ARG TF_VERSION=2.2.0rc3
 RUN pip install tensorflow-cpu==$TF_VERSION
 
 RUN apt-get update && apt-get install -y sudo rsync
diff --git a/tools/run_gpu_tests.sh b/tools/run_gpu_tests.sh
--- a/tools/run_gpu_tests.sh
+++ b/tools/run_gpu_tests.sh
@@ -6,7 +6,7 @@ export DOCKER_BUILDKIT=1
 docker build \
        -f tools/docker/build_wheel.Dockerfile \
        --target tfa_gpu_tests \
-       --build-arg TF_VERSION=2.1.0 \
+       --build-arg TF_VERSION=2.2.0rc3 \
        --build-arg PY_VERSION=3.5 \
        -t tfa_gpu_tests ./
 docker run --rm -t -v cache_bazel:/root/.cache/bazel --gpus=all tfa_gpu_tests
Repository: tensorflow/addons

Issue:
Remove the possibility to use a float as total_steps in rectified Adam
See #1375

blocked until we release the 0.10.


Discussion:
[Link to deprecation warning.](https://github.com/tensorflow/addons/blob/7aac34c52fc8fa90a2581d3bfb086ee73ef5f61e/tensorflow_addons/optimizers/rectified_adam.py#L132)

Resolving patch:
diff --git a/tensorflow_addons/optimizers/rectified_adam.py b/tensorflow_addons/optimizers/rectified_adam.py
--- a/tensorflow_addons/optimizers/rectified_adam.py
+++ b/tensorflow_addons/optimizers/rectified_adam.py
@@ -13,8 +13,6 @@
 # limitations under the License.
 # ==============================================================================
 """Rectified Adam (RAdam) optimizer."""
-import warnings
-
 import tensorflow as tf
 from tensorflow_addons.utils.types import FloatTensorLike
 
@@ -80,10 +78,7 @@ def __init__(
         weight_decay: FloatTensorLike = 0.0,
         amsgrad: bool = False,
         sma_threshold: FloatTensorLike = 5.0,
-        # float for total_steps is here to be able to load models created before
-        # https://github.com/tensorflow/addons/pull/1375 was merged. It should be
-        # removed for Addons 0.11.
-        total_steps: Union[int, float] = 0,
+        total_steps: int = 0,
         warmup_proportion: FloatTensorLike = 0.1,
         min_lr: FloatTensorLike = 0.0,
         name: str = "RectifiedAdam",
@@ -127,15 +122,6 @@ def __init__(
         self._set_hyper("decay", self._initial_decay)
         self._set_hyper("weight_decay", weight_decay)
         self._set_hyper("sma_threshold", sma_threshold)
-        if isinstance(total_steps, float):
-            warnings.warn(
-                "The parameter `total_steps` passed to the __init__ of RectifiedAdam "
-                "is a float. This behavior is deprecated and in Addons 0.11, this "
-                "will raise an error. Use an int instead. If you get this message "
-                "when loading a model, save it again and the `total_steps` parameter "
-                "will automatically be converted to a int.",
-                DeprecationWarning,
-            )
         self._set_hyper("total_steps", int(total_steps))
         self._set_hyper("warmup_proportion", warmup_proportion)
         self._set_hyper("min_lr", min_lr)


Test patch:
Repository: tensorflow/addons

Issue:
Documentation: Focal loss example misleading
In the example of the focal loss (`tfa.losses.SigmoidFocalCrossEntropy`) the label and the prediction data seems to be swapped. 
This is misleading as `sigmoid_focal_crossentropy()` expects `y_true, y_pred`, whereas the example presents `y_pred, y_true`.

https://github.com/tensorflow/addons/blob/efa0cd9c86578f53a2eb9d80849e186bffa4d64a/tensorflow_addons/losses/focal_loss.py#L42-L47

https://github.com/tensorflow/addons/blob/efa0cd9c86578f53a2eb9d80849e186bffa4d64a/tensorflow_addons/losses/focal_loss.py#L90-L92


Discussion:
Thanks @ketograph for pointing this out! Would you want to submit a PR fixing the documentation?
Hi do we still need help on this?

Resolving patch:
diff --git a/tensorflow_addons/losses/focal_loss.py b/tensorflow_addons/losses/focal_loss.py
--- a/tensorflow_addons/losses/focal_loss.py
+++ b/tensorflow_addons/losses/focal_loss.py
@@ -40,11 +40,11 @@ class SigmoidFocalCrossEntropy(LossFunctionWrapper):
     ```python
     fl = tfa.losses.SigmoidFocalCrossEntropy()
     loss = fl(
-      [[0.97], [0.91], [0.03]],
-      [[1.0], [1.0], [0.0]])
-    print('Loss: ', loss.numpy())  # Loss: [0.00010971,
-                                            0.0032975,
-                                            0.00030611]
+      y_true = [[1.0], [1.0], [0.0]],
+      y_pred = [[0.97], [0.91], [0.03]])
+    print('Loss: ', loss.numpy())  # Loss: [6.8532745e-06,
+                                            1.9097870e-04,
+                                            2.0559824e-05]
     ```
     Usage with tf.keras API:
 


Test patch:
Repository: tensorflow/addons

Issue:
Usage with tf.keras API
https://github.com/tensorflow/addons/blob/5f618fdb92d9737da059de2a33fa606e97505398/tensorflow_addons/losses/focal_loss.py#L52-L53

The usage in `tf.keras` API example is incorrect. It should be replaced with:

```python
model = tf.keras.Model(inputs, outputs)
model.compile('sgd', loss=tfa.losses.SigmoidFocalCrossEntropy())
```


Discussion:
Thanks @abhijitmjj for pointing this out. Can you create a PR with the correction?

Resolving patch:
diff --git a/tensorflow_addons/losses/focal_loss.py b/tensorflow_addons/losses/focal_loss.py
--- a/tensorflow_addons/losses/focal_loss.py
+++ b/tensorflow_addons/losses/focal_loss.py
@@ -50,7 +50,7 @@ class SigmoidFocalCrossEntropy(LossFunctionWrapper):
 
     ```python
     model = tf.keras.Model(inputs, outputs)
-    model.compile('sgd', loss=tf.keras.losses.SigmoidFocalCrossEntropy())
+    model.compile('sgd', loss=tfa.losses.SigmoidFocalCrossEntropy())
     ```
 
     Args


Test patch:
Repository: tensorflow/addons

Issue:
Deprecate GELU 
Per https://github.com/tensorflow/community/pull/252, oonce  https://github.com/tensorflow/tensorflow/pull/41178 merges we'll need to deprecate our GELU for versions of TensorFlow that include it within core.


Discussion:
May I do this?
> May I do this?

Sure! Still need GELU to merge into core and determine which version of TF it will be included in.

Resolving patch:
diff --git a/tensorflow_addons/activations/gelu.py b/tensorflow_addons/activations/gelu.py
--- a/tensorflow_addons/activations/gelu.py
+++ b/tensorflow_addons/activations/gelu.py
@@ -43,6 +43,12 @@ def gelu(x: types.TensorLike, approximate: bool = True) -> tf.Tensor:
     Returns:
         A `Tensor`. Has the same type as `x`.
     """
+    warnings.warn(
+        "gelu activation has been migrated to core TensorFlow, "
+        "and will be deprecated in Addons 0.12.",
+        DeprecationWarning,
+    )
+
     x = tf.convert_to_tensor(x)
 
     if not options.TF_ADDONS_PY_OPS:
diff --git a/tensorflow_addons/layers/gelu.py b/tensorflow_addons/layers/gelu.py
--- a/tensorflow_addons/layers/gelu.py
+++ b/tensorflow_addons/layers/gelu.py
@@ -14,6 +14,8 @@
 # ==============================================================================
 """Implements GELU activation."""
 
+import warnings
+
 import tensorflow as tf
 from tensorflow_addons.activations import gelu
 from typeguard import typechecked
@@ -38,6 +40,12 @@ class GELU(tf.keras.layers.Layer):
 
     @typechecked
     def __init__(self, approximate: bool = True, **kwargs):
+        warnings.warn(
+            "gelu activation has been migrated to core TensorFlow, "
+            "and will be deprecated in Addons 0.12.",
+            DeprecationWarning,
+        )
+
         super().__init__(**kwargs)
         self.approximate = approximate
         self.supports_masking = True


Test patch:
Repository: tensorflow/addons

Issue:
About cos sin at tfa.image.transform_ops.angles_to_projective_transforms
Sorry, it's just my personal opinion, but if you look at this code :
https://github.com/tensorflow/addons/blob/5639b712c81e81ef2ec93d592c0ea704074a3c67/tensorflow_addons/image/transform_ops.py#L213-L268

`tf.math.cos` and `tf.math.sin` is called several times with same argument. Why not.... save this into new variables instead of calling it again and again? Something like this below :

```
cos_angles = tf.math.cos(angles)
sin_angles = tf.math.sin(angles)
```

I tried and the performance improves..... well, very little IMO, but why not?

Thanks,
Smankusors


Discussion:
You can open a small PR. Thanks

Resolving patch:
diff --git a/tensorflow_addons/image/transform_ops.py b/tensorflow_addons/image/transform_ops.py
--- a/tensorflow_addons/image/transform_ops.py
+++ b/tensorflow_addons/image/transform_ops.py
@@ -239,28 +239,24 @@ def angles_to_projective_transforms(
             angles = angle_or_angles
         else:
             raise ValueError("angles should have rank 0 or 1.")
+        cos_angles = tf.math.cos(angles)
+        sin_angles = tf.math.sin(angles)
         x_offset = (
             (image_width - 1)
-            - (
-                tf.math.cos(angles) * (image_width - 1)
-                - tf.math.sin(angles) * (image_height - 1)
-            )
+            - (cos_angles * (image_width - 1) - sin_angles * (image_height - 1))
         ) / 2.0
         y_offset = (
             (image_height - 1)
-            - (
-                tf.math.sin(angles) * (image_width - 1)
-                + tf.math.cos(angles) * (image_height - 1)
-            )
+            - (sin_angles * (image_width - 1) + cos_angles * (image_height - 1))
         ) / 2.0
         num_angles = tf.shape(angles)[0]
         return tf.concat(
             values=[
-                tf.math.cos(angles)[:, None],
-                -tf.math.sin(angles)[:, None],
+                cos_angles[:, None],
+                -sin_angles[:, None],
                 x_offset[:, None],
-                tf.math.sin(angles)[:, None],
-                tf.math.cos(angles)[:, None],
+                sin_angles[:, None],
+                cos_angles[:, None],
                 y_offset[:, None],
                 tf.zeros((num_angles, 2), tf.dtypes.float32),
             ],


Test patch:
Repository: tensorflow/addons

Issue:
Markdown parsing error in NovoGrad documentation page
The documentation page of NovoGrad has a broken Markdown link shown below:

![novograd](https://user-images.githubusercontent.com/6107926/89725432-69760a00-da4a-11ea-9c8c-1495ecf9ec69.png)

I suspect this is because only the first line of the docstring is used for the first line of the documentation page. In the docstring (shown below), the link spans multiple lines.

https://github.com/tensorflow/addons/blob/5f746971d0d9491716f2f13206299a2c45941b0c/tensorflow_addons/optimizers/novograd.py#L26-L28

The simplest solution would be to edit the docstring so that there is a one-line summary for the first line with a paper link after the newline. One example would be [LAMB's documentation page](https://www.tensorflow.org/addons/api_docs/python/tfa/optimizers/LAMB). I will create a PR with this solution.



Discussion:
Feel free to request my review after submitting the PR 😃 It's also appreciated that you can add testable docstring for it. See https://github.com/tensorflow/addons/issues/2066 for more details.

Resolving patch:
diff --git a/tensorflow_addons/optimizers/novograd.py b/tensorflow_addons/optimizers/novograd.py
--- a/tensorflow_addons/optimizers/novograd.py
+++ b/tensorflow_addons/optimizers/novograd.py
@@ -23,16 +23,16 @@
 
 @tf.keras.utils.register_keras_serializable(package="Addons")
 class NovoGrad(tf.keras.optimizers.Optimizer):
-    """The NovoGrad Optimizer was first proposed in [Stochastic Gradient
-    Methods with Layerwise Adaptvie Moments for training of Deep
-    Networks](https://arxiv.org/pdf/1905.11286.pdf)
-
-    NovoGrad is a first-order SGD-based algorithm, which computes second
-    moments per layer instead of per weight as in Adam. Compared to Adam,
-    NovoGrad takes less memory, and has been found to be more numerically
-    stable. More specifically we compute (for more information on the
-    computation please refer to this
-    [link](https://nvidia.github.io/OpenSeq2Seq/html/optimizers.html):
+    """Optimizer that implements NovoGrad.
+
+    The NovoGrad Optimizer was first proposed in [Stochastic Gradient
+    Methods with Layerwise Adaptive Moments for training of Deep
+    Networks](https://arxiv.org/pdf/1905.11286.pdf) NovoGrad is a
+    first-order SGD-based algorithm, which computes second moments per
+    layer instead of per weight as in Adam. Compared to Adam, NovoGrad
+    takes less memory, and has been found to be more numerically stable.
+    (For more information on the computation please refer to this
+    [link](https://nvidia.github.io/OpenSeq2Seq/html/optimizers.html))
 
     Second order moment = exponential moving average of Layer-wise square
     of grads:


Test patch:
Repository: tensorflow/addons

Issue:
Refactor docs directory
Per comment https://github.com/tensorflow/addons/pull/2160#discussion_r487230647, better to move these stuff out of `docs`.

- [ ] Move build tools to `tools/docs`
- [ ] Modify related documentation


Discussion:
can i take this up?
@seanpmorgan there seems to be two tasks in this issue, You can assign both of us one task or howsoever you deem fit. I would also like to work on this.

Resolving patch:
diff --git a/docs/build_docs.py b/tools/docs/build_docs.py
similarity index 100%
rename from docs/build_docs.py
rename to tools/docs/build_docs.py


Test patch:
Repository: tensorflow/addons

Issue:
Set GroupNormalization default `groups` parameter to 32.
Currently, the default `groups` parameter for [`tfa.layers.GroupNormalization`](https://www.tensorflow.org/addons/api_docs/python/tfa/layers/GroupNormalization) is set to 2.

According to the original [Group Normalization paper](https://arxiv.org/abs/1803.08494) it would make more sense to set that default to 32. For reference, here is `Table 3 (top)` from the paper:

| `groups` | 64 | 32 | 16 | 8 | 4 | 2 |1 (=LN) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ResNet-50 ImageNet validation error `%` | 24.6 | ***24.1*** | 24.6 | 24.4 | 24.6 | 24.7 | 25.3 |
| Difference to `groups=32` | 0.5|  - | 0.5 | 0.3 | 0.5 | 0.6 | 1.2 |

Note that most experiments in the paper were performed using `groups=32` as well.


Discussion:

Resolving patch:
diff --git a/tensorflow_addons/layers/normalizations.py b/tensorflow_addons/layers/normalizations.py
--- a/tensorflow_addons/layers/normalizations.py
+++ b/tensorflow_addons/layers/normalizations.py
@@ -26,6 +26,9 @@
 class GroupNormalization(tf.keras.layers.Layer):
     """Group normalization layer.
 
+    Source: "Group Normalization" (Yuxin Wu & Kaiming He, 2018)
+    https://arxiv.org/abs/1803.08494
+
     Group Normalization divides the channels into groups and computes
     within each group the mean and variance for normalization.
     Empirically, its accuracy is more stable than batch norm in a wide
@@ -46,6 +49,7 @@ class GroupNormalization(tf.keras.layers.Layer):
         groups: Integer, the number of groups for Group Normalization.
             Can be in the range [1, N] where N is the input dimension.
             The input dimension must be divisible by the number of groups.
+            Defaults to 32.
         axis: Integer, the axis that should be normalized.
         epsilon: Small float added to variance to avoid dividing by zero.
         center: If True, add offset of `beta` to normalized tensor.
@@ -66,15 +70,12 @@ class GroupNormalization(tf.keras.layers.Layer):
 
     Output shape:
         Same shape as input.
-
-    References:
-        - [Group Normalization](https://arxiv.org/abs/1803.08494)
     """
 
     @typechecked
     def __init__(
         self,
-        groups: int = 2,
+        groups: int = 32,
         axis: int = -1,
         epsilon: float = 1e-3,
         center: bool = True,


Test patch:
diff --git a/tensorflow_addons/layers/tests/normalizations_test.py b/tensorflow_addons/layers/tests/normalizations_test.py
--- a/tensorflow_addons/layers/tests/normalizations_test.py
+++ b/tensorflow_addons/layers/tests/normalizations_test.py
@@ -251,7 +251,9 @@ def test_regularizations():
     layer.build((None, 4, 4))
     assert len(layer.losses) == 2
     max_norm = tf.keras.constraints.max_norm
-    layer = GroupNormalization(gamma_constraint=max_norm, beta_constraint=max_norm)
+    layer = GroupNormalization(
+        groups=2, gamma_constraint=max_norm, beta_constraint=max_norm
+    )
     layer.build((None, 3, 4))
     assert layer.gamma.constraint == max_norm
     assert layer.beta.constraint == max_norm
Repository: tensorflow/addons

Issue:
Typos in cohens_kappa.py
This is a very minor issue but raising an issue here in order to submit a PR immediately after.

Issues:
1. `...tp be...` should be `...to be...` in a comment. See[ L45 in cohens_kappy.py](https://github.com/tensorflow/addons/blob/b13140719a3de5d1354b12cb73940acaa8dd4a79/tensorflow_addons/metrics/cohens_kappa.py#L45).
2. One row of a matrix is not aligned with the other other rows in an example. See [L80 in cohens_kappy.py](https://github.com/tensorflow/addons/blob/b13140719a3de5d1354b12cb73940acaa8dd4a79/tensorflow_addons/metrics/cohens_kappa.py#L80).


Discussion:

Resolving patch:
diff --git a/tensorflow_addons/metrics/cohens_kappa.py b/tensorflow_addons/metrics/cohens_kappa.py
--- a/tensorflow_addons/metrics/cohens_kappa.py
+++ b/tensorflow_addons/metrics/cohens_kappa.py
@@ -42,7 +42,7 @@ class CohenKappa(Metric):
             kappa statistics. A valid value is one of
             [None, 'linear', 'quadratic']. Defaults to `None`
         sparse_labels: (bool) Valid only for multi-class scenario.
-            If True, ground truth labels are expected tp be integers
+            If True, ground truth labels are expected to be integers
             and not one-hot encoded.
         regression: (bool) If set, that means the problem is being treated
             as a regression problem where you are regressing the predictions.
@@ -77,7 +77,7 @@ class CohenKappa(Metric):
     <tf.Tensor: shape=(5, 5), dtype=float32, numpy=
      array([[ 0.,  0.,  0.,  0.,  0.],
             [ 0.,  6.,  0.,  0.,  0.],
-           [ 0.,  0.,  0.,  0., 10.],
+            [ 0.,  0.,  0.,  0., 10.],
             [ 0.,  0.,  0.,  2.,  0.],
             [ 0.,  0.,  2.,  0.,  7.]], dtype=float32)>
     >>> result = metric.result()


Test patch:
Repository: tensorflow/addons

Issue:
Drop GELU for 0.13 release
https://www.tensorflow.org/api_docs/python/tf/keras/activations/gelu will be available in TF2.4. Deprecation warning is already set for our upcming 0.12 release


Discussion:
@seanpmorgan Should we maintain the backward compatibility as we're still in 0.x release? Maybe we can discuss to bump the major to 1.x along with the breaking change of optimizers in #2187.
So I don't think we **have** to keep backward compatibility while in 0.x verisons, but I do think it's a good idea. Making this one of the 1.0 breaking changes sounds good (will post an issue soon with tasks).

Resolving patch:
diff --git a/tensorflow_addons/activations/gelu.py b/tensorflow_addons/activations/gelu.py
--- a/tensorflow_addons/activations/gelu.py
+++ b/tensorflow_addons/activations/gelu.py
@@ -18,6 +18,7 @@
 import warnings
 
 from tensorflow_addons.utils import types
+from distutils.version import LooseVersion
 
 
 @tf.keras.utils.register_keras_serializable(package="Addons")
@@ -47,6 +48,9 @@ def gelu(x: types.TensorLike, approximate: bool = True) -> tf.Tensor:
     See [Gaussian Error Linear Units (GELUs)](https://arxiv.org/abs/1606.08415)
     and [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805).
 
+    Note that `approximate` will default to `False` from TensorFlow version 2.4 onwards.
+    Consider using `tf.nn.gelu` instead.
+
     Usage:
 
     >>> tfa.options.TF_ADDONS_PY_OPS = True
@@ -54,7 +58,7 @@ def gelu(x: types.TensorLike, approximate: bool = True) -> tf.Tensor:
     >>> tfa.activations.gelu(x, approximate=False)
     <tf.Tensor: shape=(3,), dtype=float32, numpy=array([-0.15865529,  0.        ,  0.8413447 ], dtype=float32)>
     >>> tfa.activations.gelu(x, approximate=True)
-    <tf.Tensor: shape=(3,), dtype=float32, numpy=array([-0.158808,  0.      ,  0.841192], dtype=float32)>
+    <tf.Tensor: shape=(3,), dtype=float32, numpy=array([-0.15880796,  0.        ,  0.841192  ], dtype=float32)>
 
     Args:
         x: A `Tensor`. Must be one of the following types:
@@ -71,7 +75,15 @@ def gelu(x: types.TensorLike, approximate: bool = True) -> tf.Tensor:
 
     x = tf.convert_to_tensor(x)
 
-    return _gelu_py(x, approximate)
+    if LooseVersion(tf.__version__) >= "2.4":
+        gelu_op = tf.nn.gelu
+        warnings.warn(
+            "Default value of `approximate` is changed from `True` to `False`"
+        )
+    else:
+        gelu_op = _gelu_py
+
+    return gelu_op(x, approximate)
 
 
 def _gelu_py(x: types.TensorLike, approximate: bool = True) -> tf.Tensor:


Test patch: