Repository: tensorflow/addons

Issue:
tfa.image.rotate Different Fill Modes
**Describe the feature and the current behavior/state.**
Currently, when an image is rotated using the `tfa.image.rotate()` function, the outer regions of the image are filled with 0's, resulting in an image with black corners usually. This is usually not preferred, especially when the user is trying to create new images via data augmentation. Images with black corners are usually not a good representation of the intended creation.

`tf.keras.preprocessing.image.random_rotation` has a really good option where you can control how exactly the blank space is filled. The following pictures shows some of the common methods and how the resulting image looks closer to the original.

![](https://cdn-images-1.medium.com/max/720/1*rG4YJyVdu28LZwkwkwC55w.jpeg)

It would be nice to see the same option added to tfa.image.rotate.

**Relevant information**
- Are you willing to contribute it (yes/no): Yes!
- Are you willing to maintain it going forward? (yes/no): Yes!
- Is there a relevant academic paper? (if so, where): No
- Is there already an implementation in another framework? (if so, where): Keras (obviously)
- Was it part of tf.contrib? (if so, where): Not that I am aware of

**Which API type would this fall under (layer, metric, optimizer, etc.)**
`tfa.image`

**Who will benefit with this feature?**
Anyone who is currently using `tfa.image.rotate` for data / image augmentation purposes and want different methods to fill the image boundaries to create better augmented images for training.

**Any other info.**
While `tf.keras.preprocessing.image.random_rotation` (assuming you are performing rotations randomly) is capable of doing the same thing, it is generally not convenient to use it in a data pipeline. For one reason, it's not really designed for tensors in the form of `(num_pictures, num_rows, num_cols, num_channels)`, the channels last format which is whats normally expected. Secondly, it expects inputs as Numpy arrays. Finally it is hard to use directly with @tf.function. Thus, I think a native implementation would be preferred.



Discussion:
It's widely used, already present in keras with numpy, I just hope it won't be too hard to implement. Contributions welcome!
shall we use the same method used in keras.
I don't have a preference on the method as long as it's fast and the gradients and the results are correct.
Ok so we need to make it from scratch.
I currently have a solution built in Python using high level Tensorflow ops for the following extensions:

* Extending the closest pixel outwards (the second picture)
* Copying the entire image along all directions (the last picture)

 I'm not sure if it would be fast enough as hoped for, especially if its used in an input-bound data pipeline, but it was quick to implement and test. I can make a PR if it's good enough for now.
It's better to have an implementation that's correct with the right API than to have a fast one. We can always do speed optimizations afterwards. So feel free to open a pull request. If you can, first do a pull request for the simplest mode, then once the first pull request is merged, do another pull request for the other mode you have. 

Having small pull requests allow us to make fast review/changes cycles.

Resolving patch:
diff --git a/tensorflow_addons/image/transform_ops.py b/tensorflow_addons/image/transform_ops.py
--- a/tensorflow_addons/image/transform_ops.py
+++ b/tensorflow_addons/image/transform_ops.py
@@ -36,6 +36,7 @@ def transform(
     images: TensorLike,
     transforms: TensorLike,
     interpolation: str = "NEAREST",
+    fill_mode: str = "CONSTANT",
     output_shape: Optional[list] = None,
     name: Optional[str] = None,
 ) -> tf.Tensor:
@@ -55,6 +56,15 @@ def transform(
         gradients are not backpropagated into transformation parameters.
       interpolation: Interpolation mode.
         Supported values: "NEAREST", "BILINEAR".
+      fill_mode: Points outside the boundaries of the input are filled according
+        to the given mode (one of `{'constant', 'reflect', 'wrap'}`).
+        - *reflect*: `(d c b a | a b c d | d c b a)`
+          The input is extended by reflecting about the edge of the last pixel.
+        - *constant*: `(k k k k | a b c d | k k k k)`
+          The input is extended by filling all values beyond the edge with the
+          same constant value k = 0.
+        - *wrap*: `(a b c d | a b c d | a b c d)`
+          The input is extended by wrapping around to the opposite edge.
       output_shape: Output dimesion after the transform, [height, width].
         If None, output is the same size as input image.
 
@@ -105,11 +115,13 @@ def transform(
                 % len(transforms.get_shape())
             )
 
+        # TODO(WindQAQ): Support "nearest" `fill_mode` and `fill_value` in TF2.4.
         output = tf.raw_ops.ImageProjectiveTransformV2(
             images=images,
             transforms=transforms,
             output_shape=output_shape,
             interpolation=interpolation.upper(),
+            fill_mode=fill_mode.upper(),
         )
         return img_utils.from_4D_image(output, original_ndims)
 
@@ -268,6 +280,7 @@ def rotate(
     images: TensorLike,
     angles: TensorLike,
     interpolation: str = "NEAREST",
+    fill_mode: str = "CONSTANT",
     name: Optional[str] = None,
 ) -> tf.Tensor:
     """Rotate image(s) counterclockwise by the passed angle(s) in radians.
@@ -282,6 +295,15 @@ def rotate(
         batch.
       interpolation: Interpolation mode. Supported values: "NEAREST",
         "BILINEAR".
+      fill_mode: Points outside the boundaries of the input are filled according
+        to the given mode (one of `{'constant', 'reflect', 'wrap'}`).
+        - *reflect*: `(d c b a | a b c d | d c b a)`
+          The input is extended by reflecting about the edge of the last pixel.
+        - *constant*: `(k k k k | a b c d | k k k k)`
+          The input is extended by filling all values beyond the edge with the
+          same constant value k = 0.
+        - *wrap*: `(a b c d | a b c d | a b c d)`
+          The input is extended by wrapping around to the opposite edge.
       name: The name of the op.
 
     Returns:
@@ -304,6 +326,7 @@ def rotate(
             images,
             angles_to_projective_transforms(angles, image_height, image_width),
             interpolation=interpolation,
+            fill_mode=fill_mode,
         )
         return img_utils.from_4D_image(output, original_ndims)
 
diff --git a/tensorflow_addons/image/translate_ops.py b/tensorflow_addons/image/translate_ops.py
--- a/tensorflow_addons/image/translate_ops.py
+++ b/tensorflow_addons/image/translate_ops.py
@@ -75,26 +75,36 @@ def translate(
     images: TensorLike,
     translations: TensorLike,
     interpolation: str = "NEAREST",
+    fill_mode: str = "CONSTANT",
     name: Optional[str] = None,
 ) -> tf.Tensor:
     """Translate image(s) by the passed vectors(s).
 
     Args:
       images: A tensor of shape
-          `(num_images, num_rows, num_columns, num_channels)` (NHWC),
-          `(num_rows, num_columns, num_channels)` (HWC), or
-          `(num_rows, num_columns)` (HW). The rank must be statically known (the
-          shape is not `TensorShape(None)`).
+        `(num_images, num_rows, num_columns, num_channels)` (NHWC),
+        `(num_rows, num_columns, num_channels)` (HWC), or
+        `(num_rows, num_columns)` (HW). The rank must be statically known (the
+        shape is not `TensorShape(None)`).
       translations: A vector representing `[dx, dy]` or (if `images` has rank 4)
-          a matrix of length num_images, with a `[dx, dy]` vector for each image
-          in the batch.
+        a matrix of length num_images, with a `[dx, dy]` vector for each image
+        in the batch.
       interpolation: Interpolation mode. Supported values: "NEAREST",
-          "BILINEAR".
+        "BILINEAR".
+      fill_mode: Points outside the boundaries of the input are filled according
+        to the given mode (one of `{'constant', 'reflect', 'wrap'}`).
+        - *reflect*: `(d c b a | a b c d | d c b a)`
+          The input is extended by reflecting about the edge of the last pixel.
+        - *constant*: `(k k k k | a b c d | k k k k)`
+          The input is extended by filling all values beyond the edge with the
+          same constant value k = 0.
+        - *wrap*: `(a b c d | a b c d | a b c d)`
+          The input is extended by wrapping around to the opposite edge.
       name: The name of the op.
     Returns:
       Image(s) with the same type and shape as `images`, translated by the
-      given vector(s). Empty space due to the translation will be filled with
-      zeros.
+        given vector(s). Empty space due to the translation will be filled with
+        zeros.
     Raises:
       TypeError: If `images` is an invalid type.
     """
@@ -103,6 +113,7 @@ def translate(
             images,
             translations_to_projective_transforms(translations),
             interpolation=interpolation,
+            fill_mode=fill_mode,
         )
 
 


Test patch:
diff --git a/tensorflow_addons/image/tests/transform_ops_test.py b/tensorflow_addons/image/tests/transform_ops_test.py
--- a/tensorflow_addons/image/tests/transform_ops_test.py
+++ b/tensorflow_addons/image/tests/transform_ops_test.py
@@ -68,6 +68,62 @@ def test_extreme_projective_transform(dtype):
     )
 
 
+@pytest.mark.with_device(["cpu", "gpu"])
+@pytest.mark.usefixtures("maybe_run_functions_eagerly")
+@pytest.mark.parametrize("dtype", _DTYPES)
+def test_transform_constant_fill_mode(dtype):
+    image = tf.constant(
+        [[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11], [12, 13, 14, 15]], dtype=dtype
+    )
+    expected = np.asarray(
+        [[0, 0, 1, 2], [0, 4, 5, 6], [0, 8, 9, 10], [0, 12, 13, 14]],
+        dtype=dtype.as_numpy_dtype,
+    )
+    # Translate right by 1 (the transformation matrix is always inverted,
+    # hence the -1).
+    translation = tf.constant([1, 0, -1, 0, 1, 0, 0, 0], dtype=tf.float32)
+    image_transformed = transform_ops.transform(
+        image, translation, fill_mode="constant"
+    )
+    np.testing.assert_equal(image_transformed.numpy(), expected)
+
+
+@pytest.mark.with_device(["cpu", "gpu"])
+@pytest.mark.usefixtures("maybe_run_functions_eagerly")
+@pytest.mark.parametrize("dtype", _DTYPES)
+def test_transform_reflect_fill_mode(dtype):
+    image = tf.constant(
+        [[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11], [12, 13, 14, 15]], dtype=dtype
+    )
+    expected = np.asarray(
+        [[0, 0, 1, 2], [4, 4, 5, 6], [8, 8, 9, 10], [12, 12, 13, 14]],
+        dtype=dtype.as_numpy_dtype,
+    )
+    # Translate right by 1 (the transformation matrix is always inverted,
+    # hence the -1).
+    translation = tf.constant([1, 0, -1, 0, 1, 0, 0, 0], dtype=tf.float32)
+    image_transformed = transform_ops.transform(image, translation, fill_mode="reflect")
+    np.testing.assert_equal(image_transformed.numpy(), expected)
+
+
+@pytest.mark.with_device(["cpu", "gpu"])
+@pytest.mark.usefixtures("maybe_run_functions_eagerly")
+@pytest.mark.parametrize("dtype", _DTYPES)
+def test_transform_wrap_fill_mode(dtype):
+    image = tf.constant(
+        [[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11], [12, 13, 14, 15]], dtype=dtype
+    )
+    expected = np.asarray(
+        [[3, 0, 1, 2], [7, 4, 5, 6], [11, 8, 9, 10], [15, 12, 13, 14]],
+        dtype=dtype.as_numpy_dtype,
+    )
+    # Translate right by 1 (the transformation matrix is always inverted,
+    # hence the -1).
+    translation = tf.constant([1, 0, -1, 0, 1, 0, 0, 0], dtype=tf.float32)
+    image_transformed = transform_ops.transform(image, translation, fill_mode="wrap")
+    np.testing.assert_equal(image_transformed.numpy(), expected)
+
+
 @pytest.mark.usefixtures("maybe_run_functions_eagerly")
 def test_transform_static_output_shape():
     image = tf.constant([[1.0, 2.0], [3.0, 4.0]])