Repository: keras-team/keras

Issue:
clipnorm doesn't work with Embedding
I'm getting a Traceback every time "clipnorm" is used in NN with Embedding layer.
Here is a simple script where the problem is obvious:

``` python
import numpy as np
from keras.layers import Input, Embedding
from keras.optimizers import Adam
from keras.models import Model

input_layer = Input(shape = (1,) )

embedding = Embedding(input_dim = 1,
                      output_dim = 1)(input_layer)

model = Model(input = input_layer, output = embedding)

model.compile(optimizer = Adam(clipnorm = 1.0), loss = 'mse')

X = np.array([[1]])
Y = np.array([[[0.5]]])
model.fit(X, Y, nb_epoch = 1)
```

Failure:

``` shell
I tensorflow/core/common_runtime/gpu/gpu_device.cc:867] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:01:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:867] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX TITAN X, pci bus id: 0000:02:00.0)
Traceback (most recent call last):
  File "./clipnorm-bug.py", line 20, in <module>
    model.fit(X, Y, nb_epoch = 1)
  File "/usr/local/lib/python3.5/dist-packages/keras/engine/training.py", line 1079, in fit
    self._make_train_function()
  File "/usr/local/lib/python3.5/dist-packages/keras/engine/training.py", line 696, in _make_train_function
    self.total_loss)
  File "/usr/local/lib/python3.5/dist-packages/keras/optimizers.py", line 379, in get_updates
    grads = self.get_gradients(loss, params)
  File "/usr/local/lib/python3.5/dist-packages/keras/optimizers.py", line 71, in get_gradients
    grads = [clip_norm(g, self.clipnorm, norm) for g in grads]
  File "/usr/local/lib/python3.5/dist-packages/keras/optimizers.py", line 71, in <listcomp>
    grads = [clip_norm(g, self.clipnorm, norm) for g in grads]
  File "/usr/local/lib/python3.5/dist-packages/keras/optimizers.py", line 9, in clip_norm
    g = K.switch(n >= c, g * c / n, g)
TypeError: unsupported operand type(s) for *: 'IndexedSlices' and 'float'
```

Keras version is 1.1.0, TensorFlow is 0.10rc

clipvalue on the other hand works fine.



Discussion:
I have the same issue

The way tensorflow handles scalar multiplication with `IndexedSlices` is: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/math_ops.py#L458-L467

@fchollet Should we convert the scalar to tensor 
- here or,
- move `clip_norm` to the backend from https://github.com/fchollet/keras/blob/master/keras/optimizers.py and convert it there

I can make a PR on the desired behavior

I got bit by this as well. Would be great if someone could fix it!
I too just ran into this issue with `clipnorm` and embeddings. I also confirm that `clipvalue` does work.
@fchollet - This seems like a serious issue regarding clipnorm as it's very useful for training RNN's, which often employs an Embedding layer. Could you give some guidance on how to fix this issue?
@kracwarlock Did you manage to find a solution?
I think I switched back to keras/tf versions where this was working
my project does not need this anymore but will try to write a fix if I get time
@PiranjaF you can try out the branch I have PR'd above (https://github.com/kracwarlock/keras/tree/fix-3859)
it works for me
Thanks, it works!
I made some changes to the PR. Can you re-check? The latest one also works for me locally.
+1, I also got bit by this as well. @kracwarlock i'll try out your patch in a bit and let you know if it works on my setup, thanks for contributing.

EDIT: it works for me, thanks!
@kracwarlock Sorry about the delay, I didn't have a free GPU for testing. The latest one works for me as well.
Hi, 
When will this patch be included in the main Keras branch and the pip release?
Thanks a lot.
What is a current status of this bug? Clip norm is very important for training RNN's and this bug kills the opportunity of using it in models with RNN's and embedded layer. 
agreed. has anyone been able to check if the bug is present in `tf.contrib.keras` (https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/keras)?

Resolving patch:
diff --git a/keras/optimizers.py b/keras/optimizers.py
--- a/keras/optimizers.py
+++ b/keras/optimizers.py
@@ -1,5 +1,6 @@
 from __future__ import absolute_import
 import six
+import copy
 from six.moves import zip
 
 from . import backend as K
@@ -11,7 +12,30 @@
 
 
 def clip_norm(g, c, n):
-    if c > 0:
+    if c <= 0:  # if clipnorm == 0 no need to add ops to the graph
+        return g
+
+    # tf require using a special op to multiply IndexedSliced by scalar
+    if K.backend() == 'tensorflow':
+        condition = n >= c
+        then_expression = tf.scalar_mul(c / n, g)
+        else_expression = g
+
+        # saving the shape to avoid converting sparse tensor to dense
+        if isinstance(then_expression, tf.Tensor):
+            g_shape = copy.copy(then_expression.get_shape())
+        elif isinstance(then_expression, tf.IndexedSlices):
+            g_shape = copy.copy(then_expression.dense_shape)
+        if condition.dtype != tf.bool:
+            condition = tf.cast(condition, 'bool')
+        g = tf.cond(condition,
+                    lambda: then_expression,
+                    lambda: else_expression)
+        if isinstance(then_expression, tf.Tensor):
+            g.set_shape(g_shape)
+        elif isinstance(then_expression, tf.IndexedSlices):
+            g._dense_shape = g_shape
+    else:
         g = K.switch(K.greater_equal(n, c), g * c / n, g)
     return g
 


Test patch:
diff --git a/tests/integration_tests/test_temporal_data_tasks.py b/tests/integration_tests/test_temporal_data_tasks.py
--- a/tests/integration_tests/test_temporal_data_tasks.py
+++ b/tests/integration_tests/test_temporal_data_tasks.py
@@ -6,7 +6,8 @@
 from keras.utils.test_utils import get_test_data, keras_test
 from keras.utils.np_utils import to_categorical
 from keras.models import Sequential
-from keras import layers
+from keras import layers, optimizers
+import keras.backend as K
 import keras
 
 
@@ -204,5 +205,14 @@ def test_masked_temporal():
     ground_truth = -np.log(0.5)
     assert(np.abs(history.history['loss'][-1] - ground_truth) < 0.06)
 
+
+@pytest.mark.skipif(K.backend() != 'tensorflow', reason='Requires TF backend')
+@keras_test
+def test_embedding_with_clipnorm():
+    model = Sequential()
+    model.add(layers.Embedding(input_dim=1, output_dim=1))
+    model.compile(optimizer=optimizers.SGD(clipnorm=0.1), loss='mse')
+    model.fit(np.array([[0]]), np.array([[[0.5]]]), epochs=1)
+
 if __name__ == '__main__':
     pytest.main([__file__])