Repository: sql-machine-learning/elasticdl

Issue:
Support convert categorical features to SparseTensor for embedding in Keras
## Why we need to convert Keras inputs to sparse tensor?
In the real dataset, feature values of samples may be missing and we want to ignore the missing value during transformation. So we need to use `tf.SparseTensor` to represent the input. For example:

|education | marital-status|
| --------- | ------------|
| Master | Divorced
|    | Never-married
| Bachelor |    |

The input tensor from dataset for `tf.keras.layers.Input` is
```python
{
    "education":[["Master"], [""], ["Bachelor"]],
    "marital-status":[["Divorced"], ["Never-married"], [""]]
}
```
In the case, we may want to ingore the empty string and convert other values to zero-based integer ids for embedding. So we need to convert the inputs to `tf.SparseTensor` like:
```python
{
    "education":SparseTensor(
       indices=tf.Tensor([[0, 0], [2, 0]], shape=(2, 2), dtype=int64), 
       values=tf.Tensor(["Master", "Bachelor"], shape=(2,), dtype=string),
       dense_shape=tf.Tensor([3,1], shape=(2,), dtype=int64)
   )
   "marital-status":SparseTensor(
       indices=tf.Tensor([[0, 0], [1, 0]], shape=(2, 2), dtype=int64), 
       values=tf.Tensor(["Divorced", "Never-married"], shape=(2,), dtype=string),
       dense_shape=tf.Tensor([3,1], shape=(2,), dtype=int64)
   )
```
Then, we can use keras preprocessing layers proposed in the [RFC](https://github.com/tensorflow/community/pull/188) like `Lookup` and `Hash` to convert the values of `tf.SparseTensor` to zero-based integer ids. And we can feed them to the customized [SparseEmbedding](https://github.com/sql-machine-learning/elasticdl/blob/develop/elasticdl/python/keras/layers/sparse_embedding.py) with combiner in ElasticDL. For example, we use `Lookup` with education vocabulary ["Master", "Bachelor"] and marital-status vocabulary ["Divorced", "Never-married"] and can get

```python
{
    "education":SparseTensor(
       indices=tf.Tensor([[0, 0], [2, 0]], shape=(2, 2), dtype=int64), 
       values=tf.Tensor([0, 1], shape=(2,), dtype=string),
       dense_shape=tf.Tensor([3,1], shape=(2,), dtype=int64)
   )
   "marital-status":SparseTensor(
       indices=tf.Tensor([[0, 0], [1, 0]], shape=(2, 2), dtype=int64), 
       values=tf.Tensor([0, 1], shape=(2,), dtype=int64),
       dense_shape=tf.Tensor([3,1], shape=(2,), dtype=int64)
   )
```

When the number of categorical features is very large, we may need to concatenate the integer ids of multiple categorical features before embedding and we have described the feature in the [issue](https://github.com/tensorflow/tensorflow/issues/37521).  For example, we add the ids of marital-status with 3 to avoid the conflicts with education ids and then concatenate the two `SparseTensor` to a `SparseTensor`:
```python
SparseTensor(
       indices=tf.Tensor([[0, 0], [2, 0], [0, 1], [1, 1]], shape=(2, 2), dtype=int64), 
       values=tf.Tensor([0, 1, 3, 4], shape=(2,), dtype=int64),
       dense_shape=tf.Tensor([3, 2], shape=(2,), dtype=int64)
   )
```


## The problem of filling the missing value with a default integer id before embedding.
Besides ignoring the missing value, we also can fill the missing value with a default integer id during transformation.
The vocabulary of education is ["Master", "Bachelor"] and the id of missing value is 2.
The vocabulary of martial-status is ["Divorced", "Never-married"] and the id of missing value is 2.
After lookup, we can get the transformation result:
```python
{"education": [[0], [2], [1]], "marital-status": [[0], [1], [2]]}
```
Then we may want to concatenate the integer ids like the [issue](https://github.com/tensorflow/tensorflow/issues/37521).  Before concatenating, we add the ids of marital-status with 3 to avoid the conflicts with education ids. So, the concatenation result is
```python
 [[0, 3], [2, 4], [1, 5]]
```
Then we feed the result into  `tf.keras.layers.Embedding` and use `tf.reduce_sum` to combine the embedding output to 3x2 dense tensor. The logic of embedding and `tf.reduce_sum` is the same as `SparseEmbedding` with the sum combiner. But, we add the embedding values of missing values in the result. **The combined result of each sample may be bigger than using sparse tensor without missing values. The embedding values of missing values may be noise for training.**

For example:
Suppose that the embedding table is 6x2 matrix:
```python
 [[0.1, 0.1], [0.2, 0.2], [0.3, 0.3], [0.4, 0.4], [0.5, 0.5], [0.6, 0.6]]
```
If we feed the above concatenated sparse tensor into `SparseEmbedding` with sum combiner, the output is
```python
[[0.5, 0.5],  [0.5, 0.5], [0.2, 0.2]]
```
If we feed the concatenated tensor into `tf.keras.layers.Embedding` and  use `tf.reduce_sum` to combine, the output is
```python
[[0.5, 0.5],  [0.8, 0.8], [0.8, 0.8]]
```



Discussion:

Resolving patch:
diff --git a/elasticdl_preprocessing/layers/to_sparse.py b/elasticdl_preprocessing/layers/to_sparse.py
new file mode 100644
--- /dev/null
+++ b/elasticdl_preprocessing/layers/to_sparse.py
@@ -0,0 +1,66 @@
+import tensorflow as tf
+
+
+class ToSparse(tf.keras.layers.Layer):
+    """Converts a `Tensor` to a `SparseTensor`, dropping ignore_value cells.
+    If the input is already a `SparseTensor`, just return it.
+
+    Example :
+    ```python
+        layer = ToSparse()
+        inp = tf.constant([["A", ""], ["B", "C"]], tf.string)
+        out = layer(inp)
+    ```
+    The expected output is
+    ```
+    tf.SparseTensor(
+            indices=np.array([[0, 0], [1, 0], [1, 1]]),
+            values=np.array(["A", "B", "C"]),
+            dense_shape=(2, 2),
+        )
+    ```
+
+    Arguments:
+        ignore_value: Entries in inputs equal to this value will be
+            absent from the output `SparseTensor`. If `None`, default value of
+            inputs dtype will be used ('' for `str`, -1 for `int`).
+
+    Input shape: A numeric or string `Tensor` of shape
+        `[batch_size, d1, ..., dm]`
+
+    Output shape: An `SparseTensor` with the same shape as inputs
+    """
+
+    def __init__(self, ignore_value=None):
+        super(ToSparse, self).__init__()
+        self.ignore_value = ignore_value
+
+    def call(self, inputs):
+        if isinstance(inputs, tf.SparseTensor):
+            return inputs
+
+        ignore_value = self.ignore_value
+        if ignore_value is None:
+            if inputs.dtype == tf.string:
+                ignore_value = ""
+            elif inputs.dtype.is_integer:
+                ignore_value = -1
+            else:
+                ignore_value = 0.0
+        ignore_value = tf.cast(ignore_value, inputs.dtype)
+        indices = tf.where(tf.not_equal(inputs, ignore_value))
+        values = tf.gather_nd(inputs, indices)
+        dense_shape = tf.shape(inputs, out_type=tf.int64)
+        return tf.SparseTensor(
+            indices=indices, values=values, dense_shape=dense_shape
+        )
+
+    def compute_output_shape(self, input_shape):
+        return input_shape
+
+    def get_config(self):
+        config = {
+            "ignore_value": self.ignore_value,
+        }
+        base_config = super(ToSparse, self).get_config()
+        return dict(list(base_config.items()) + list(config.items()))


Test patch:
diff --git a/elasticdl_preprocessing/tests/to_sparse_test.py b/elasticdl_preprocessing/tests/to_sparse_test.py
new file mode 100644
--- /dev/null
+++ b/elasticdl_preprocessing/tests/to_sparse_test.py
@@ -0,0 +1,47 @@
+import unittest
+
+import numpy as np
+import tensorflow as tf
+
+from elasticdl_preprocessing.layers.to_sparse import ToSparse
+from elasticdl_preprocessing.tests.test_utils import sparse_tensor_equal
+
+
+class ToSparseTest(unittest.TestCase):
+    def test_to_sparse(self):
+        layer = ToSparse()
+        inp = tf.constant([["A", ""], ["B", "C"]], tf.string)
+        output = layer.call(inp)
+        expected_out = tf.SparseTensor(
+            indices=np.array([[0, 0], [1, 0], [1, 1]]),
+            values=np.array(["A", "B", "C"]),
+            dense_shape=(2, 2),
+        )
+        self.assertTrue(sparse_tensor_equal(output, expected_out))
+
+        layer = ToSparse()
+        inp = tf.constant([[12, -1], [45, 78]], tf.int64)
+        output = layer.call(inp)
+        expected_out = tf.SparseTensor(
+            indices=np.array([[0, 0], [1, 0], [1, 1]]),
+            values=np.array([12, 45, 78]),
+            dense_shape=(2, 2),
+        )
+        self.assertTrue(sparse_tensor_equal(output, expected_out))
+
+    def test_model_with_to_sparse(self):
+        inputs = tf.keras.Input(shape=(1,), dtype=tf.int32)
+        sparse_inputs = ToSparse(ignore_value=-1)(inputs)
+        model = tf.keras.Model(inputs=inputs, outputs=sparse_inputs)
+        out = model.call(tf.constant([[1], [-1], [2], [3]]))
+
+        expect_out = tf.SparseTensor(
+            indices=tf.constant([[0, 0], [2, 0], [3, 0]], dtype=tf.int64),
+            values=tf.constant([1, 2, 3], dtype=tf.int32),
+            dense_shape=(4, 1),
+        )
+        self.assertTrue(sparse_tensor_equal(out, expect_out))
+
+
+if __name__ == "__main__":
+    unittest.main()