Repository: Theano/Theano

Issue:
Get rid of the hack in dnn_conv to use the gradient ops
Timings I did of the convolution indicate that the forward op is much faster when workmem='small' than the gradient ops.  And by much faster I mean 0.5 s vs 10 s for shapes of (64, 3, 227, 227) and (96, 3, 11, 11).

For tiny sizes, like (1, 3, 8, 8) and (4, 3, 2, 2), its 5.1e-4 vs 4.5e-4 which is a bit slower for the plain forward of, but not by much.

We should disable the hack that tries to use the gradient ops by default and let people that want max speed in all cases use the meta-optimizer for convolution.

cuDNN convolution strong performance regression
6a63844 renders the automatic introduction of the cuDNN convolution partly ineffective. It prevents `GpuDnnConvGradI` being introduced for the gradient wrt. inputs. See the performance difference before and after that commit, using cuDNN v2:

**Before:**

```
$ SKIP=legacy,fft,convnet,meta,gemm python pylearn2_benchmark.py 1
Using gpu device 0: Tesla K40c

CONFIG: input = 3 x 128 x 128 * ker = 3 x 96 x 11 x 11 ( bs = 128 , stride = 1 )
(auto) theano.sandbox.cuda.dnn.GpuDnnConv          ==> fprop         ==>      78
(auto) theano.sandbox.cuda.dnn.GpuDnnConv          ==> bprop inputs  ==>     111
(auto) theano.sandbox.cuda.dnn.GpuDnnConv          ==> bprop weights ==>     105

(manual conv) theano.sandbox.cuda.dnn.GpuDnnConv   ==> fprop         ==>      79
(manual conv) theano.sandbox.cuda.dnn.GpuDnnConv   ==> bprop inputs  ==>     111
(manual conv) theano.sandbox.cuda.dnn.GpuDnnConv   ==> bprop weights ==>     105
```

**After:**

```
$ SKIP=legacy,fft,convnet,meta,gemm python pylearn2_benchmark.py 1
Using gpu device 0: Tesla K40c (CNMeM is disabled)

CONFIG: input = 3 x 128 x 128 * ker = 3 x 96 x 11 x 11 ( bs = 128 , stride = 1 )
(auto) theano.sandbox.cuda.dnn.GpuDnnConv          ==> fprop         ==>      78
(auto) theano.sandbox.cuda.dnn.GpuDnnConv          ==> bprop inputs  ==>    1069
(auto) theano.sandbox.cuda.dnn.GpuDnnConv          ==> bprop weights ==>     106

(manual conv) theano.sandbox.cuda.dnn.GpuDnnConv   ==> fprop         ==>      79
(manual conv) theano.sandbox.cuda.dnn.GpuDnnConv   ==> bprop inputs  ==>     112
(manual conv) theano.sandbox.cuda.dnn.GpuDnnConv   ==> bprop weights ==>     106
```

Inspecting the graph shows that `GpuDnnConv` is being used for the backward pass after the commit, while `GpuDnnConvGradI` was used before that commit.

What was the rationale for changing it?

There are three possible solutions:
1. Removing the `version()` check again
2. Changing the `version()` check to include cuDNN v2
3. Changing the check to only use `GpuDnnConvGradI` when `direction_hint == 'bprop inputs'` (this was proposed before, but then we decided to wait until #2665 was finished, which seemed to be due shortly at that time)

fix dnn_conv() direction_hint check
Commit 05dc20d introduced a check for `direction_hint` in `dnn_conv()` that is too strict. `dnn_conv()` is regularly called with `direction_hint='bprop inputs'`, and the meta-optimizer may also call it with `direction_hint='forward!'` (the latter was supposed to change, but in discussing it we decided to wait until #2665 is finished, which seemed to be within a few weeks at that time).



Discussion:
If it don't speed up anymore, I agree with your direction. But for cudnn
v1, we should choose to remove support of it or warm about it being not
optimized anymore or keep the old code for it.

I'm prefer removing the code and warm at dnn init about v1 if the user have
it.
Le 18 mars 2015 13:25, "abergeron" notifications@github.com a écrit :

> Timings I did of the convolution indicate that the forward op is much
> faster when workmem='small' than the gradient ops. And by much faster I
> mean 0.5 s vs 10 s for shapes of (64, 3, 227, 227) and (96, 3, 11, 11).
> 
> For tiny sizes, like (1, 3, 8, 8) and (4, 3, 2, 2), its 5.1e-4 vs 4.5e-4
> which is a bit slower for the plain forward of, but not by much.
> 
> We should disable the hack that tries to use the gradient ops by default
> and let people that want max speed in all cases use the meta-optimizer for
> convolution.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/Theano/Theano/issues/2644.

I would agree with that change.  However I want to note that we still get a speedup from this hack in the case where workmem='none'.  I would not consider this as significant since it's not the default anymore.

If someone really wants to use workmem='none' and wants the maximum speed for that case, there is always the meta optimizer.

> Timings I did of the convolution indicate that the forward op is much faster when workmem='small' than the gradient ops.

Hmm, not in my case (input `(64, 1, 115, 80)`, kernel `(32, 1, 8, 6)`). The gradient ops are still a bit faster than the forward op.

> We should disable the hack that tries to use the gradient ops by default

Wait, I think I'm not sure which "hack" you are referring to. `dnn_conv` has the `direction_hint` parameter, which [is used by the optimizer](https://github.com/Theano/Theano/blob/master/theano/sandbox/cuda/dnn.py#L1491-1494) to choose the gradient ops when gradients are computed, and which [is used by the meta-optimizer](https://github.com/Theano/Theano/blob/master/theano/sandbox/cuda/dnn.py#L1513-1527) to try two different variants (either the gradient op that would have been chosen by default, or the forward op).
If you call `dnn_conv()` directly, it will use the forward op for a valid convolution, and `DnnConvGradI` for a full convolution unless you specify `direction_hint='forward!'` to override this choice. Do you suggest to change this behavior, i.e., [this `if` clause](https://github.com/Theano/Theano/blob/master/theano/sandbox/cuda/dnn.py#L685-686)? Or do you suggest to change the optimizer to always replace `GpuConv` with the forward op?

My timings using the script [convnet-benchmarks/theano/pylearn2_benchmark.py](https://github.com/soumith/convnet-benchmarks/blob/master/theano/pylearn2_benchmark.py) (you can just download this single file and it will work out-of-the-box):

```
$ THEANO_FLAGS=metaopt.verbose=1 SKIP=legacy,gemm,fft,convnet,dnn python pylearn2_benchmark.py i1x115x80,k32x8x6,b64
Using gpu device 0: GeForce GTX 780 Ti

CONFIG: input = 1 x 115 x 80 * ker = 1 x 32 x 8 x 6 ( bs = 64 , stride = 1 )
ConvMetaOptimizer meta-optimizing GpuConv{valid, (1, 1), None, (6, 8), True, (1, 80, 115), (6, 8)}(GpuFromHost.0, GpuFromHost.0) (5 choices):
* local_conv_fft_full: not applicable
* local_conv_fft_valid: 0.0155 sec
* local_conv_dnn: 0.0022948 sec
* local_conv_gemm: 0.0050981 sec
* local_conv_dnn_alternative: 0.0025442 sec
= local_conv_dnn
(experimental) meta-optimizer                      ==> fprop         ==>       2
ConvMetaOptimizer meta-optimizing GpuConv{full, (1, 1), None, (6, 8), True, (32, 75, 108), (6, 8)}(GpuFromHost.0, GpuFromHost.0) (5 choices):
* local_conv_fft_full: 0.01887 sec
* local_conv_fft_valid: not applicable
* local_conv_dnn: 0.00348 sec
* local_conv_gemm: 0.004663 sec
* local_conv_dnn_alternative: 0.039531 sec
= local_conv_dnn
(experimental) meta-optimizer                      ==> bprop inputs  ==>       3
ConvMetaOptimizer meta-optimizing GpuConv{valid, (1, 1), None, (75, 108), False, (64, 80, 115), (75, 108)}(GpuFromHost.0, GpuFromHost.0) (5 choices):
* local_conv_fft_full: not applicable
* local_conv_fft_valid: 0.014567 sec
* local_conv_dnn: 0.00739 sec
* local_conv_gemm: 0.012213 sec
* local_conv_dnn_alternative: 0.068126 sec
= local_conv_dnn
(experimental) meta-optimizer                      ==> bprop weights ==>       3
```

`local_conv_dnn_alternative` is the time for using the forward op instead of the gradient op (except for the very first case, where it's the time for using the gradient wrt. weights instead of the forward op). It's always slower. (This is using cuDNN v2 final, Theano b1963ef, and the default workmem setting.)

@abergeron: Can you please post your results for `i3x227x227,k96x11x11,b64`? My current GPU is too small for that.

> Timings I did of the convolution indicate that the forward op is much faster when workmem='small' than the gradient ops. And by much faster I mean 0.5 s vs 10 s for shapes of (64, 3, 227, 227) and (96, 3, 11, 11).

Another idea: Can you profile that? Could it be because you replaced the `cp_if_negative_strides` by `gpu_contiguous` so it's copying large matrices around when using the gradient ops?

My can't run that script as-is either with the shapes I've given.  I made another script which only times the dnn convolution and it can run with these shapes on my gpu.

This is the script: https://gist.github.com/abergeron/2a6d34710459183c416f#file-time_conv-py

For i(64, 1, 115, 80) k(96, 1, 8, 6) 'bc01' (the shape you used for the timings above) I get these times:

```
Using gpu device 0: GeForce GTX 750 Ti
valid      : ( 1000 loops ): 0.00597714304924 s
full  hack : ( 10   loops ): 0.251130986214 s
full nohack: ( 1000 loops ): 0.00767396616936 s
```

This seems somewhat far from the times you have (even if the GPU differ the relative differences should not vary so much).  Can you run my script on you GPU?  You can change the sizes to try the same as the pylearn2_benchmark ones.

If we get a significant difference, the problem that remains is: why?  Is it because I do my timings incorrectly?  Is it because the meta-optimizer times incorrectly?  Are we actually timing the same thing?

> why? Is it because I do my timings incorrectly? Is it because the meta-optimizer times incorrectly?Are we actually timing the same thing?

Your timings are good and the meta-optimizer is good, but we're not timing the same operation!
What I've been timing with the meta-optimizer was the _backward pass_ for a valid convolution of `(64, 1, 115, 80)` with `(32, 1, 8, 6)`, i.e., a full convolution of the gradient wrt. the output with the weights to obtain the gradient wrt. the inputs. In shapes, that's a full convolution of `(64, 32, 115-8+1, 80-6+1)` with `(1, 32, 8, 6)`, for which I get:

```
valid      : ( 10 loops ): 0.224610900879 s
full  hack : ( 100 loops ): 0.0179100012779 s
full nohack: ( 10 loops ): 0.301650500298 s
```

So for these shapes it's important to not use the forward pass algorithm, and it seemed that this was true for all full convolutions (note that there was no `workmem='small'` option at the time I implemented this). What we can do now is either:
1. Change `dnn_conv` so it only uses `GpuDnnConvGradI` when `direction_hint='bprop inputs'` (and it's a full non-strided convolution, as before); change the `local_conv_dnn_alternative` optimizer accordingly, or
2. Figure out from the shape which algorithm would probably be faster and use this one by default (if the shape is given)

I can easily do the first option if you're okay with it. This way everything would be the same for valid convolutions using `conv2d()`, and performance could improve or degrade for full convolutions, depending on the shape.
We _could_ behave differently for `workmem='none'`, but that would be a bit awkward (it would introduce a spaghetti-like dependency of `local_conv_dnn_alternative` on the implementation of `dnn_conv()`). It's better to ask people to rely on the meta-optimizer if performance is an issue.

By the way, note that the "hack" with the `direction_hint` attribute and argument will be dropped with #2505, as the placeholder ops will carry direction information naturally.

/edit: Note that for a valid convolution of the shape timed above, it would be much faster to do a `GpuDnnConvGradI` with subsequent cropping than to do a `GpuDnnConv` or a `GpuDnnConvGradW`. That's something the meta-optimizer doesn't try yet, but it should (although such a drastic reduction in the number of channels is probably not occurring often in a network).
/edit2: With regard to the last sentence, there already is a TODO for that in #2072.

Since we have someone working on #2505 which should be ready real soon, I think we should just wait for it.

Also, making the MetaOptimizer try all three ops sounds like a good idea, but perhaps it shouldn't rely on dnn_conv() at all.

> Since we have someone working on #2505 which should be ready real soon,

Cool! Please let me know if I can help reviewing when it's there.

> I think we should just wait for it.

Agreed.

> but perhaps it shouldn't rely on dnn_conv() at all.

This depends on how #2505 turns out. At the time, it seemed a good idea to have the complexity of setting up the "fake" convolutions in some central place, as it helped for direct calls of `dnn_conv()` as well.




Resolving patch:
diff --git a/theano/sandbox/cuda/dnn.py b/theano/sandbox/cuda/dnn.py
--- a/theano/sandbox/cuda/dnn.py
+++ b/theano/sandbox/cuda/dnn.py
@@ -1115,7 +1115,7 @@ def dnn_conv(img, kerns, border_mode='valid', subsample=(1, 1),
         If border_mode is 'valid', subsample is (1,1) and direction_hint is
         'bprop weights', it will use GpuDnnConvGradW.
         If border_mode is 'full', subsample is (1,1) and direction_hint is
-        *not* 'forward!', it will use GpuDnnConvGradI.
+        'bprop inputs', it will use GpuDnnConvGradI.
         This parameter is used internally by graph optimizers and may be
         removed at any time without a deprecation period. You have been warned.
     :param workmem: *deprecated*, use param algo instead
@@ -1138,7 +1138,7 @@ def dnn_conv(img, kerns, border_mode='valid', subsample=(1, 1),
         algo = workmem
 
     # Ensure the value of direction_hint is supported
-    assert direction_hint in [None, 'bprop weights', 'forward']
+    assert direction_hint in [None, 'bprop weights', 'bprop inputs', 'forward']
 
     fgraph = getattr(img, 'fgraph', None) or getattr(kerns, 'fgraph', None)
     if (border_mode == 'valid' and subsample == (1, 1) and
@@ -1161,12 +1161,10 @@ def dnn_conv(img, kerns, border_mode='valid', subsample=(1, 1),
         return as_cuda_ndarray_variable(conv.dimshuffle(1, 0, 2, 3))
 
     elif (border_mode == 'full' and subsample == (1, 1) and
-            direction_hint != 'forward!' and version() == -1):
-        # Special case: In CuDNN v1, we can be faster by using GpuDnnConvGradI
-        # to compute the full convolution as the backward pass of a valid
-        # convolution. We just need to set up a suitable 'fake' valid
-        # convolution.
-        img = gpu_contiguous(img)  # cudnn v1 and v2 rc3 need contiguous data
+            direction_hint == 'bprop inputs'):
+        # Special case: We are asked to use GpuDnnConvGradI. We need to set
+        # up a suitable 'fake' convolution to compute the gradient for.
+        img = gpu_contiguous(img)
         kerns = gpu_contiguous(kerns.dimshuffle(1, 0, 2, 3))
         conv_mode = 'cross' if conv_mode == 'conv' else 'conv'
         shape2 = shape_i(img, 2, fgraph) + shape_i(kerns, 2, fgraph) - 1
@@ -2049,8 +2047,11 @@ def local_conv_dnn_alternative(node):
             direction_hint = node.op.direction_hint
             if border_mode == 'full':
                 # for a full convolution, try using the forward pass instead
-                # of the backward pass wrt. inputs
-                direction_hint = 'forward!'
+                # of the backward pass wrt. inputs and vice versa
+                if direction_hint == 'bprop inputs':
+                    direction_hint = 'forward'
+                else:
+                    direction_hint = 'bprop inputs'
             elif border_mode == 'valid':
                 # for a valid convolution, try using the backward pass wrt.
                 # weights instead of the forward pass and vice versa


Test patch: