Repository: Theano/Theano

Issue:
Return correct gradient in ARange with float dtype (and floatX-typed gradient when dtype is discrete)
Currently, the [implementation](https://github.com/Theano/Theano/blob/372ae95ef9aae402b7f426d159585d26556eb0a1/theano/tensor/basic.py#L5395-L5397) re-uses the dtype of the inptus, which is not correct.

I think there is a helper function to get the right dtype (floatX if dtype in discrete_dtypes, else dtype) somewhere.


Discussion:
Wait, that formula is not even correct when the dtype is not discrete!
We should figure out the right formula (including the case where step < 0), or at least return `GradNotImplemented()` in the mean time.
When the output is continuous:
- I think that the gradient for the "stop" should be 0, as it does not change the result most of the time when changed by epsilon.
- The gradient wrt "start" should be the sum of output gradients
- The gradient wrt "step" should be `sum(dout * arange(n_steps+1))`, where `n_steps` is the actual number of steps taken during the forward.
What is the consequence of this bug?
Wrong results (0 gradient) when taking gradients through an ARange with a float dtype.

Resolving patch:
diff --git a/theano/tensor/basic.py b/theano/tensor/basic.py
--- a/theano/tensor/basic.py
+++ b/theano/tensor/basic.py
@@ -5475,6 +5475,7 @@ def make_node(self, start, stop, step):
 
         inputs = [start, stop, step]
         outputs = [tensor(self.dtype, (False,))]
+
         return Apply(self, inputs, outputs)
 
     @theano.configparser.change_flags(warn_float64='ignore')
@@ -5526,7 +5527,7 @@ def connection_pattern(self, node):
 
         return [[True], [False], [True]]
 
-    def grad(self, inputs, grads):
+    def L_op(self, inputs, outputs, grads):
         start, stop, step = inputs
         gz, = grads
         # start and step affect the output values
@@ -5534,9 +5535,17 @@ def grad(self, inputs, grads):
         # no gradient through them
         # stop does not affect the output values,
         # just the output shape, so it is disconnected
-        return [start.zeros_like(),
-                DisconnectedType()(),
-                step.zeros_like()]
+
+        if (self.dtype in discrete_dtypes) and (step > 0):
+            return [start.zeros_like(),
+                    DisconnectedType()(),
+                    step.zeros_like()]
+        else:
+            num_steps_taken = outputs[0].shape[0]  # (stop-start)/step
+            return [gz.sum(dtype=config.floatX),
+                    DisconnectedType()(),
+                    (gz * arange(num_steps_taken, dtype=self.dtype)).sum(
+                        dtype=config.floatX)]
 
     def R_op(self, inputs, eval_points):
         return [None]


Test patch:
diff --git a/theano/tensor/tests/test_basic.py b/theano/tensor/tests/test_basic.py
--- a/theano/tensor/tests/test_basic.py
+++ b/theano/tensor/tests/test_basic.py
@@ -5911,6 +5911,20 @@ def test_Op_integers(self):
         assert np.all(f(10, 2, 2) == np.arange(10, 2, 2))
         assert np.all(f(0, 0, 1) == np.arange(0, 0, 1))
 
+    def test_grads(self):
+        def f(start, stop, step):
+            return ARange(start.type.dtype)(start, stop, step)
+
+        rng = np.random.RandomState(utt.fetch_seed())
+        # Due to the random projection, we should not use the exact
+        # point that shage the shape of the output.
+        for start, stop, step in [(0, 4.9, 1),
+                                  (1, 5.1, 0.5)]:
+            utt.verify_grad(f, [np.float32(start),
+                                np.float32(stop),
+                                np.float32(step)],
+                            rng=rng)
+
     def test_integers(self):
         # Test arange constructor, on integer outputs
         start, stop, step = iscalars('start', 'stop', 'step')