Repository: Theano/Theano

Issue:
AbstractConv2d does not implement R_op
While trying to calculate a Hessian vector product of a loss function involving convolutions, I came across the following exception which told me to report this to theano:

```
NotImplementedError: AbstractConv2d{border_mode=(0, 0), subsample=(1, 1),
filter_flip=True, imshp=(None, 1, 28, 28), kshp=(32, 1, 5, 5)} of class
AbstractConv2d does not implement R_op. If this is a theano op, write to the
theano-dev mailing list for assistance. If it is your own op, implement the R_op method.
```

I will be happy to help resolving this, however I are not familiar with theanos convolutions.



Discussion:
Thanks for the report, and for volunteering to help!
Since convolutions (and gradient of convolutions) are linear operators, we can use the same formula as for the `dot` operation: the R operator of `L(a, b)`, evaluated at `c` for `a` and `d` for `b` is `L(c, b) + L(a, d)`.

So the skeleton implementation or R_op for the `AbstractConv2d` would be something like:

``` python
def R_op(self, inputs, eval_points):
    return [self(eval_points[0], inputs[1]) + self(inputs[0], eval_points[1])]
```

It gets a bit more complex when you have to handle non-provided evaluation points (`None`), but I think you can simply return `[None]` if `eval_points[0]` or `eval_points[1]` is `None`.

For the gradient Ops, it is almost the same, but there is a third input to the Op, which is the shape of its output. It should not have any evaluation point provided, so we can just reuse the input. The gist would be:

``` python
def R_op(self, inputs, eval_points):
    return [self(eval_points[0], inputs[1], inputs[2]) + self(inputs[0], eval_points[1], inputs[2])]
```

Don't hesitate to ask more questions if the above is not clear.
Thanks,

The old convolution in theano/tensor/nnet/conv.py have an R_op()
implementation. You can look at it.

On Tue, Jan 19, 2016 at 11:28 AM, Pascal Lamblin notifications@github.com
wrote:

> Thanks for the report, and for volunteering to help!
> Since convolutions (and gradient of convolutions) are linear operators, we
> can use the same formula as for the dot operation: the R operator of L(a,
> b), evaluated at c for a and d for b is L(c, b) + L(a, d).
> 
> So the skeleton implementation or R_op for the AbstractConv2d would be
> something like:
> 
> def R_op(self, inputs, eval_points):
>     return [self(eval_points[0], inputs[1]) + self(inputs[0], eval_points[1])]
> 
> It gets a bit more complex when you have to handle non-provided evaluation
> points (None), but I think you can simply return [None] if eval_points[0]
> or eval_points[1] is None.
> 
> For the gradient Ops, it is almost the same, but there is a third input to
> the Op, which is the shape of its output. It should not have any evaluation
> point provided, so we can just reuse the input. The gist would be:
> 
> def R_op(self, inputs, eval_points):
>     return [self(eval_points[0], inputs[1], inputs[2]) + self(inputs[0], eval_points[1], inputs[2])]
> 
> Don't hesitate to ask more questions if the above is not clear.
> Thanks,
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/Theano/Theano/issues/3888#issuecomment-172907133.


Resolving patch:
diff --git a/theano/tensor/nnet/abstract_conv.py b/theano/tensor/nnet/abstract_conv.py
--- a/theano/tensor/nnet/abstract_conv.py
+++ b/theano/tensor/nnet/abstract_conv.py
@@ -308,6 +308,17 @@ def perform(self, node, inp, out_):
             'Did you exclude both "conv_dnn" and "conv_gemm" from '
             'the optimizer? Is cudnn available and does the GPU support it?')
 
+    def R_op(self, inputs, eval_points):
+        rval = None
+        if eval_points[0] is not None:
+            rval = self.make_node(eval_points[0], inputs[1]).outputs[0]
+        if eval_points[1] is not None:
+            if rval is None:
+                rval = self.make_node(inputs[0], eval_points[1]).outputs[0]
+            else:
+                rval += self.make_node(inputs[0], eval_points[1]).outputs[0]
+        return [rval]
+
     def grad(self, inp, grads):
         bottom, weights = inp
         top, = grads


Test patch:
diff --git a/theano/tests/test_rop.py b/theano/tests/test_rop.py
--- a/theano/tests/test_rop.py
+++ b/theano/tests/test_rop.py
@@ -22,7 +22,7 @@
 from theano.gradient import grad_undefined
 from theano.tests.unittest_tools import SkipTest
 from theano.tensor.signal.pool import Pool
-from theano.tensor.nnet import conv
+from theano.tensor.nnet import conv, conv2d
 
 '''
 Special Op created to test what happens when you have one op that is not
@@ -260,50 +260,51 @@ def test_rebroadcast(self):
             (1,))
 
     def test_conv(self):
-        for border_mode in ['valid', 'full']:
-            image_shape = (2, 2, 4, 5)
-            filter_shape = (2, 2, 2, 3)
-            image_dim = len(image_shape)
-            filter_dim = len(filter_shape)
-            input      = tensor.TensorType(
-                theano.config.floatX,
-                [False] * image_dim)(name='input')
-            filters    = tensor.TensorType(
-                theano.config.floatX,
-                [False] * filter_dim)(name='filter')
-            ev_input   = tensor.TensorType(
-                theano.config.floatX,
-                [False] * image_dim)(name='ev_input')
-            ev_filters = tensor.TensorType(
-                theano.config.floatX,
-                [False] * filter_dim)(name='ev_filters')
-
-            def sym_conv2d(input, filters):
-                return conv.conv2d(input, filters, border_mode=border_mode)
-            output = sym_conv2d(input, filters).flatten()
-            yv = tensor.Rop(output, [input, filters], [ev_input, ev_filters])
-            rop_f = function([input, filters, ev_input, ev_filters],
-                             yv, on_unused_input='ignore')
-            sy, _ = theano.scan(
-                lambda i, y, x1, x2, v1, v2:
-                    (tensor.grad(y[i], x1) * v1).sum() + \
-                    (tensor.grad(y[i], x2) * v2).sum(),
-                                sequences=tensor.arange(output.shape[0]),
-                                non_sequences=[output, input, filters,
-                                               ev_input, ev_filters])
-            scan_f = function([input, filters, ev_input, ev_filters], sy,
-                              on_unused_input='ignore')
-
-            dtype = theano.config.floatX
-            image_data = numpy.random.random(image_shape).astype(dtype)
-            filter_data = numpy.random.random(filter_shape).astype(dtype)
-            ev_image_data = numpy.random.random(image_shape).astype(dtype)
-            ev_filter_data = numpy.random.random(filter_shape).astype(dtype)
-            v1 = rop_f(image_data, filter_data, ev_image_data,
-                       ev_filter_data)
-            v2 = scan_f(image_data, filter_data, ev_image_data,
+        for conv_op in [conv.conv2d, conv2d]:
+            for border_mode in ['valid', 'full']:
+                image_shape = (2, 2, 4, 5)
+                filter_shape = (2, 2, 2, 3)
+                image_dim = len(image_shape)
+                filter_dim = len(filter_shape)
+                input      = tensor.TensorType(
+                    theano.config.floatX,
+                    [False] * image_dim)(name='input')
+                filters    = tensor.TensorType(
+                    theano.config.floatX,
+                    [False] * filter_dim)(name='filter')
+                ev_input   = tensor.TensorType(
+                    theano.config.floatX,
+                    [False] * image_dim)(name='ev_input')
+                ev_filters = tensor.TensorType(
+                    theano.config.floatX,
+                    [False] * filter_dim)(name='ev_filters')
+
+                def sym_conv2d(input, filters):
+                    return conv_op(input, filters, border_mode=border_mode)
+                output = sym_conv2d(input, filters).flatten()
+                yv = tensor.Rop(output, [input, filters], [ev_input, ev_filters])
+                rop_f = function([input, filters, ev_input, ev_filters],
+                                yv, on_unused_input='ignore')
+                sy, _ = theano.scan(
+                    lambda i, y, x1, x2, v1, v2:
+                        (tensor.grad(y[i], x1) * v1).sum() + \
+                        (tensor.grad(y[i], x2) * v2).sum(),
+                                    sequences=tensor.arange(output.shape[0]),
+                                    non_sequences=[output, input, filters,
+                                                ev_input, ev_filters])
+                scan_f = function([input, filters, ev_input, ev_filters], sy,
+                                on_unused_input='ignore')
+
+                dtype = theano.config.floatX
+                image_data = numpy.random.random(image_shape).astype(dtype)
+                filter_data = numpy.random.random(filter_shape).astype(dtype)
+                ev_image_data = numpy.random.random(image_shape).astype(dtype)
+                ev_filter_data = numpy.random.random(filter_shape).astype(dtype)
+                v1 = rop_f(image_data, filter_data, ev_image_data,
                         ev_filter_data)
-            assert numpy.allclose(v1, v2), ("Rop mismatch: %s %s" %
+                v2 = scan_f(image_data, filter_data, ev_image_data,
+                            ev_filter_data)
+                assert numpy.allclose(v1, v2), ("Rop mismatch: %s %s" %
                                             (v1, v2))
 
     def test_join(self):