Repository: keras-team/keras

Issue:
Tests crash for the shape inference
Hello all,

When I tried to run some scripts after updating to the latest version of Theano and Keras I had some errors.

I maybe didn't notice a change on my side but I tried to run the tests and it seems the shape inference is not behaving nicely:

``` python
Using gpu device 0: GeForce GTX 980 (CNMeM is disabled)
......FF.........
======================================================================
FAIL: test_shape_inference.test_Convolution1D
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/thomas/anaconda3/envs/anapy27/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/thomas/libraries/forks/keras/tests/test_shape_inference.py", line 75, in test_Convolution1D
    check_layer_output_shape(layer, input_data)
  File "/home/thomas/libraries/forks/keras/tests/test_shape_inference.py", line 18, in check_layer_output_shape
    assert output.shape[1:] == expected_output_shape, "output shape: {} expected output shape: {}".format(output.shape[1:], ex
pected_output_shape)                                                                                                         
AssertionError: output shape: (2, 1) expected output shape: (3, 1)

======================================================================
FAIL: test_shape_inference.test_Convolution2D
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/thomas/anaconda3/envs/anapy27/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/thomas/libraries/forks/keras/tests/test_shape_inference.py", line 91, in test_Convolution2D
    check_layer_output_shape(layer, input_data)
  File "/home/thomas/libraries/forks/keras/tests/test_shape_inference.py", line 18, in check_layer_output_shape
    assert output.shape[1:] == expected_output_shape, "output shape: {} expected output shape: {}".format(output.shape[1:], ex
pected_output_shape)                                                                                                         
AssertionError: output shape: (1, 2, 2) expected output shape: (1, 3, 3)

----------------------------------------------------------------------
Ran 17 tests in 7.296s

FAILED (failures=2)
```

I tried to digg in and I made my example work but the tests are still failing.

This is my reproductible example:

``` python
from sklearn.feature_extraction.image import extract_patches_2d
import theano
import keras


import numpy as np
np.random.seed(1337)

from keras.models import Graph
from keras.layers.core import Dense, Flatten
from keras.layers.convolutional import Convolution1D, MaxPooling1D

len_ts_y = 60
sample = 1000
Fs = 8000
f = 5
x_bug = np.arange(sample)

y_test = np.sin(2 * np.pi * f * x_bug / Fs)
y_test = y_test+np.random.normal(0, 0.2, sample)

data_patched = extract_patches_2d(y_test[:,None], (len_ts_y+1,1))
y_train = data_patched[:,:,-1]
endog_train = data_patched[:,-len_ts_y-1:-1,-1]

model = Graph()
model.add_input(name='endog', input_shape=(len_ts_y,1))

model.add_node(Convolution1D(nb_filter=4,
                            filter_length=2,
                            border_mode="same",
                            activation="relu",
                            subsample_length=1,
                            ),
                            name='conv_1', input='endog')
model.add_node(Flatten(), name="flat", input='conv_1')
model.add_node(Dense(32, activation="sigmoid"), name="Dense", input='flat')
model.add_node(Dense(1, activation="softplus"), name='last_dense', input='Dense')
model.add_output(name='output', input='last_dense')
model.compile(optimizer='sgd', loss={'output':'mse'})

model.fit({'endog': endog_train[:-100,:, None], 'output': y_train[:-100,-1]},
          validation_data={'endog':endog_train[-100:,:, None], 'output': y_train[-100:,-1]},
          batch_size=128,
          nb_epoch=4)
```

I changed the size of the `output_shape` in the `conv_output_length` function of `convolutionnal.py` to make the above example work :

``` python
def conv_output_length(input_length, filter_size, border_mode, stride):
    if input_length is None:
        return None
    assert border_mode in {'same', 'valid'}
    if border_mode == 'same':
        output_length = input_length
    elif border_mode == 'valid':
        output_length = input_length - filter_size + 1
    return ((output_length + stride - 1) // stride) - 1
```

Does anyone is having the same errors?

Thank you!



Discussion:
I can't reproduce, tests pass here and on Travis. And your script runs perfectly normally on my laptop.

> AssertionError: output shape: (1, 2, 2) expected output shape: (1, 3, 3)

My best guess is that your version of Theano has a conv2d op that behaves abnormally. Just upgrade Theano to latest master branch and you should be fine. 

I checked theano version inside the test and it outputs the right one (0.7.0.dev-24ace59). 
I just cleared the compile directory in my .theano folder to make sure I have a clean version but the two tests still fail.
I'm using CUDA 7.0 and cuDNN v2, could it be related?

Do you have the same issue on CPU?

It might be a bug of Theano's conv2d on GPU with a specific version of cuDNN. Who knows...

@fchollet Can you look into #1274 

It's working on CPU. Because I have CUDA 6.5 and CUDA 7.0 installed to test tensorflow, I'll try to reinstall cuDNN v3 and rerun the tests. 
Right after the release off Keras supporting Tensorflow and Theano, I was able to run both backends.
Tx for the tips, I'll keep you posted!

No luck, I tried the four combinations of CUDA/cuDNN and it still fails. I also tried to checkout older versions of the 2 repos (Theano + Keras) back to 2 weeks from now without success. If someone has a working config for both Theano and Tensorflow working on gpu on Ubuntu 14.04, I would be interested to have some advices :smile: (versions and environment vars). I will try to build a new config from scratch in Docker tomorow.
I'm still able to run my scripts and the tests using `border_mode='valid'` though...
@farizrahman4u sorry to open another issue, I'm not sure if these 2 are related and I didn't want to mix Windows and Ubuntu. If it turns out it's the same problem don't hesitate to reference this one.

I just finished to test the same scripts with both Theano  (CUDA 7.0 + cuDNN v3) and Tensorflow (CUDA 7.0 + cuDNN v2) in Docker containers and everything works as expected. I don't know what happened but it isn't a Keras related issue.

Really sorry to repoen the issue. I thought it was a config bug but I changed the size of the filters with the new config the last time I used the scripts.
To have a clean environment, I used [this container](https://hub.docker.com/r/tboquet/scikenacuthe7hc3/) built with CUDA 7.5 and cuDNN v3 and the error arise only when the filters have an even size when using 1d convolutions with Graph models (including the example above). The imdb_cnn_lstm example is running fine so I will rewrite it using Graph. The tests also fail and I will digg in the code tomorow.
I'm not sure where to look to point out the bug so if you have any suggestion I will be happy to help.

Thank you!

**Update**: the bug happens only when using cuDNN.
When bypassing it in the `theano_backend` everything work as expected:

``` python
    if _on_gpu() and dnn.dnn_available() and False:
        if border_mode == 'same':
            assert(strides == (1, 1))
            np_kernel = kernel.eval()
            pad_x = (np_kernel.shape[2] - strides[0]) // 2
            pad_y = (np_kernel.shape[3] - strides[1]) // 2
            conv_out = dnn.dnn_conv(img=x,
                                    kerns=kernel,
                                    border_mode='full')
        else:
            conv_out = dnn.dnn_conv(img=x,
                                    kerns=kernel,
                                    border_mode=border_mode,
                                    subsample=strides)
```

With a simple script I tried to higlight the bug:

``` python
import numpy as np
import theano
import itertools
from keras.layers.convolutional import Convolution1D, Convolution2D

# Convolution 1D
l_in = (100, 16)
x = np.random.randn(20, 100, 16).astype(theano.config.floatX)

for layer, mode, f_size in itertools.product([Convolution1D],
                                                         ['valid', 'same'],
                                                         [3, 4, 7, 8]):
    try:
        l_1 = layer(input_shape=l_in, nb_filter=10, filter_length=f_size, border_mode=mode)
        out = theano.function([l_1.get_input()], l_1.get_output())
        print("conv1D, {}, {}, {}".format(mode, f_size, out(x).shape))
    except Exception as e:
        print(e)


# Convolution 2d
l_in = (3, 100, 100)
x = np.random.randn(20, 3, 100, 100).astype(theano.config.floatX)

for layer, mode, f_size in itertools.product([Convolution2D],
                                                         ['valid', 'same'],
                                                         [3, 4, 7, 8]):
    try:
        l_1 = layer(10, f_size, f_size, border_mode=mode, input_shape=l_in)
        out = theano.function([l_1.get_input()], l_1.get_output())
        print("conv2D, {}, {}, {}".format(mode, f_size, out(x).shape))
    except Exception as e:
        print(e)
```

 and I have this output when using cuDNN:

``` python
Using Theano backend.
conv1D, valid, 3, (20, 98, 10)
conv1D, valid, 4, (20, 97, 10)
conv1D, valid, 7, (20, 94, 10)
conv1D, valid, 8, (20, 93, 10)
conv1D, same, 3, (20, 100, 10)
conv1D, same, 4, (20, 99, 10)
conv1D, same, 7, (20, 100, 10)
conv1D, same, 8, (20, 99, 10)
conv2D, valid, 3, (20, 10, 98, 98)
conv2D, valid, 4, (20, 10, 97, 97)
conv2D, valid, 7, (20, 10, 94, 94)
conv2D, valid, 8, (20, 10, 93, 93)
conv2D, same, 3, (20, 10, 100, 100)
conv2D, same, 4, (20, 10, 99, 99)
conv2D, same, 7, (20, 10, 100, 100)
conv2D, same, 8, (20, 10, 99, 99)
```

Results without cuDNN:

``` python
conv1D, valid, 3, (20, 98, 10)
conv1D, valid, 4, (20, 97, 10)
conv1D, valid, 7, (20, 94, 10)
conv1D, valid, 8, (20, 93, 10)
conv1D, same, 3, (20, 100, 10)
conv1D, same, 4, (20, 100, 10)
conv1D, same, 7, (20, 100, 10)
conv1D, same, 8, (20, 100, 10)
conv2D, valid, 3, (20, 10, 98, 98)
conv2D, valid, 4, (20, 10, 97, 97)
conv2D, valid, 7, (20, 10, 94, 94)
conv2D, valid, 8, (20, 10, 93, 93)
conv2D, same, 3, (20, 10, 100, 100)
conv2D, same, 4, (20, 10, 100, 100)
conv2D, same, 7, (20, 10, 100, 100)
conv2D, same, 8, (20, 10, 100, 100)
```

@fchollet not sure if you could reproduce the bug with cuDNN?

I don't readily have access to a GPU, so I won't be able to repro.

Does this happen with every cuDNN version? Specifically, does this happen with R4?


Resolving patch:
diff --git a/keras/backend/theano_backend.py b/keras/backend/theano_backend.py
--- a/keras/backend/theano_backend.py
+++ b/keras/backend/theano_backend.py
@@ -548,12 +548,14 @@ def conv2d(x, kernel, strides=(1, 1), border_mode='valid', dim_ordering='th',
     if _on_gpu() and dnn.dnn_available():
         if border_mode == 'same':
             assert(strides == (1, 1))
-            np_kernel = kernel.eval()
-            pad_x = (np_kernel.shape[2] - strides[0]) // 2
-            pad_y = (np_kernel.shape[3] - strides[1]) // 2
             conv_out = dnn.dnn_conv(img=x,
                                     kerns=kernel,
-                                    border_mode=(pad_x, pad_y))
+                                    border_mode='full')
+            shift_x = (kernel.shape[2] - 1) // 2
+            shift_y = (kernel.shape[3] - 1) // 2
+            conv_out = conv_out[:, :,
+                                shift_x:x.shape[2] + shift_x,
+                                shift_y:x.shape[3] + shift_y]
         else:
             conv_out = dnn.dnn_conv(img=x,
                                     kerns=kernel,


Test patch: