Repository: Theano/Theano

Issue:
Bad shape computation on Windows/GPU for test_pool2d.
Hello ! I encounter an extremely strange bug on Windows (8.1 64 bits, Python 2.7.13) with the following test:

```
set THEANO_FLAGS=cuda.enabled=false,mode=DebugMode

nosetests -v theano/gpuarray/tests/test_pool.py:test_pool2d > out.log 2>&1
```

And I get the following output in out.log:

```
Using cuDNN version 5110 on context None
Mapped name None to device cuda: GeForce 840M (0000:0A:00.0)
theano.gpuarray.tests.test_pool.test_pool2d ... ERROR

======================================================================
ERROR: theano.gpuarray.tests.test_pool.test_pool2d
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\donnees\programmes\Anaconda3\envs\python2\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
  File "C:\Users\HPPC\mila\dev\git\theano\theano\gpuarray\tests\test_pool.py", line 116, in test_pool2d
    assert np.allclose(f(), f2()), (shp, ws, st, pad, mode, ignore_border)
  File "C:\Users\HPPC\mila\dev\git\theano\theano\compile\function_module.py", line 884, in __call__
    self.fn() if output_subset is None else\
  File "C:\Users\HPPC\mila\dev\git\theano\theano\compile\debugmode.py", line 2305, in deco
    return f()
  File "C:\Users\HPPC\mila\dev\git\theano\theano\compile\debugmode.py", line 2230, in f
    r_vals)
  File "C:\Users\HPPC\mila\dev\git\theano\theano\compile\debugmode.py", line 1078, in _find_bad_optimizations0
    new_graph=new_graph_str)
BadOptimization: BadOptimization Error 
  Variable: id 639243936 HostFromGpu(gpuarray).0
  Op HostFromGpu(gpuarray)(GpuPool{ignore_border=True, mode='max', ndim=2}.0)
  Value Type: <type 'numpy.ndarray'>
  Old Value shape, dtype, strides: (30L, 2L, 13L, 13L) float64 (2704L, 1352L, 104L, 8L)

  Old Value:  [[[[-0.96755897 -0.29244356  0.78700533 ...,  0.00378702 -0.10711377
     0.93837834]
   [ 0.47694224  0.87576524  0.92534936 ...,  0.99758935  0.40448142
    -0.66445048]
   [ 0.93300339  0.94034438  0.12368468 ...,  0.4604226   0.88514465
     0.72758153]
   ..., 
   [ 0.72613404  0.14583407  0.48289082 ...,  0.51951041  0.41615083
     0.80047109]
   [ 0.1070701   0.48948974 -0.11433198 ...,  0.9628016   0.58135613
    -0.80634723]
   [-0.47439399  0.98734073  0.13435195 ..., -0.09583944  0.47218597
    -0.29492752]]

  [[-0.2879721   0.6462485   0.30175485 ..., -0.64172283 -0.34572719
    -0.0173356 ]
   [ 0.64046216  0.73982045  0.77623422 ...,  0.4313929   0.26451732
     0.88228857]
   [ 0.72350992  0.7645468   0.62122458 ...,  0.86078268  0.42171147
     0.71549608]
   ..., 
   [ 0 ...
  New Value shape, dtype, strides: (30L, 2L, 1L, 1L) float64 (16L, 8L, 8L, 8L)

  New Value:  [[[[ 0.998626  ]]

  [[ 0.9971792 ]]]


 [[[ 0.99828528]]

  [[ 0.99375607]]]


 [[[ 0.99905548]]

  [[ 0.99934141]]]


 [[[ 0.99698984]]

  [[ 0.9965656 ]]]


 [[[ 0.99479379]]

  [[ 0.99761483]]]


 [[[ 0.99653428]]

  [[ 0.99685067]]]


 [[[ 0.99972085]]

  [[ 0.99980636]]]


 [[[ 0.99089935]]

  [[ 0.99980006]]]


 [[[ 0.99894417]]

  [[ 0.99617586]]]


 [[[ 0.99873954]]

  [[ 0.99375722]]]


 [[[ 0.99506163]]

  [[ 0.99737363]]]


 [[[ 0.99923836]]

  [[ 0.99895733]]]


 [[[ 0.99981227]]

  [[ 0.99822001]]]


 [[[ 0.99583118]]

  [[ 0.99254772]]]


 [[[ 0.99781263]]

  [[ 0.99732236]]]


 [[[ 0.99822466]]

  [[ 0.98472125]]]


 [[[ 0.99769789]]

  [[ 0.99986654]]]


 [[[ 0.98523309]]

  [[ 0.99811911]]]


 [[[ 0.99794538]]

  [[ 0.99928265]]]


 [[[ 0.99972087]]

  [[ 0.99923829]]]


 ...
  Reason:  GraphToGPU
  Old Graph:
  Pool{ignore_border=True, mode='max', ndim=2} [id A] ''   
   |a [id B]
   |TensorConstant{(2L,) of 2} [id C]
   |TensorConstant{(2L,) of 2} [id C]
   |TensorConstant{(2L,) of 1} [id D]

  New Graph:
  HostFromGpu(gpuarray) [id E] ''   
   |GpuPool{ignore_border=True, mode='max', ndim=2} [id F] ''   
     |GpuContiguous [id G] ''   
     | |GpuFromHost<None> [id H] ''   
     |   |a [id B]
     |TensorConstant{(2L,) of 2} [id C]
     |TensorConstant{(2L,) of 2} [id C]
     |TensorConstant{(2L,) of 1} [id D]


Hint: relax the tolerance by setting tensor.cmp_sloppy=1
  or even tensor.cmp_sloppy=2 for less-strict comparison


----------------------------------------------------------------------
Ran 1 test in 1.731s

FAILED (errors=1)
```

The shape of the pool result should be (30,2,13,13) but HostFromGpu returns a result with shape (30,2,1,1). After investigations, it seems that it's because the shape of the output is not computed correctly in GPU in `theano/gpuarray/pool.c`. The shape is computed with this macro: https://github.com/Theano/Theano/blob/master/theano/gpuarray/pool.c#L211
```
// output shape for a given input padded shape, window shape and stride
#define OUTPUT_DIMS(in_dim, ws, st)                       \
  (IGNORE_BORDER ? (in_dim - ws)/st + 1 :                 \
   (st > ws ? (in_dim - 1)/st + 1 :                       \
   std::max<size_t>(0, (in_dim - 1 - ws + st)/st) + 1))
```
that is called here: https://github.com/Theano/Theano/blob/master/theano/gpuarray/pool.c#L247
```
  for (int i = 0; i < ndims; i++) {
    w[i] = *((npy_intp*)PyArray_GETPTR1(ws, i));
    s[i] = *((npy_intp*)PyArray_GETPTR1(stride, i));
    p[i] = *((npy_intp*)PyArray_GETPTR1(pad, i));
    z_dims[2 + i] = OUTPUT_DIMS(x_dims[2 + i] + 2*p[i], w[i], s[i]);
```

I have added many fprintf to check how computation are done, and I finally find a bad compute with the following fprintf:
````
    fprintf(stderr, "GPU 24/%d == %d\n", s[i], 24/s[i]);
    fprintf(stderr, "GPU 24%%%d == %d\n", s[i], 24%s[i]);
    fprintf(stderr, "GPU 24+%d == %d\n", s[i], 24+s[i]);
    fprintf(stderr, "GPU 24-%d == %d\n", s[i], 24-s[i]);
    fprintf(stderr, "GPU 24*%d == %d\n", s[i], 24*s[i]);
    fprintf(stderr, "GPU TEST rs %d ? %d, ws %d ? %d, st %d ? %d\n",
        x_dims[2 + i] + 2*p[i],
        x_dims[2 + i] + 2*p[i] == 26,
        w[i],
        w[i] == 2,
        s[i],
        s[i] == 2
    );
````

which prints this:
```
GPU 24/2 == 0
GPU 24%2 == 24
GPU 24+2 == 26
GPU 24-2 == 22
GPU 24*2 == 48
GPU TEST rs 26 ? 0, ws 2 ? 0, st 2 ? 0
```

So, we remark that the integer division, the integer modulo and the equal comparison return completely bad results. As the integer division is used in the macro to compute output shapes, it seems that the problem come from here.

I have thought the problem was caused by my compiler:
```
(python2) C:\Users\HPPC\mila\dev\cpp>g++ -v
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=C:/donnees/programmes/Anaconda3/envs/python2/Library/mingw-w
64/bin/../lib/gcc/x86_64-w64-mingw32/5.3.0/lto-wrapper.exe
Target: x86_64-w64-mingw32
Configured with: ../gcc-5.3.0/configure --prefix=/mingw64 --with-local-prefix=/m
ingw64/local --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32 --target=x86_6
4-w64-mingw32 --with-native-system-header-dir=/mingw64/x86_64-w64-mingw32/includ
e --libexecdir=/mingw64/lib --with-gxx-include-dir=/mingw64/include/c++/5.3.0 --
enable-bootstrap --with-arch=x86-64 --with-tune=generic --enable-languages=c,lto
,c++,objc,obj-c++,fortran,ada --enable-shared --enable-static --enable-libatomic
 --enable-threads=posix --enable-graphite --enable-fully-dynamic-string --enable
-libstdcxx-time=yes --disable-libstdcxx-pch --disable-libstdcxx-debug --enable-v
ersion-specific-runtime-libs --disable-isl-version-check --enable-lto --enable-l
ibgomp --disable-multilib --enable-checking=release --disable-rpath --disable-wi
n32-registry --disable-nls --disable-werror --disable-symvers --with-libiconv --
with-system-zlib --with-gmp=/mingw64 --with-mpfr=/mingw64 --with-mpc=/mingw64 --
with-isl=/mingw64 --with-pkgversion='Rev5, Built by MSYS2 project' --with-bugurl
=https://sourceforge.net/projects/msys2 --with-gnu-as --with-gnu-ld
Thread model: posix
gcc version 5.3.0 (Rev5, Built by MSYS2 project)
```

But when I compile a simple test:
```
#include <cstdio>
int main() {
    size_t s[2] = {2,10};
    for (int i = 0; i < 2; ++i) {
        fprintf(stderr, "24 / %d = %d\n", s[i], 24/s[i]);
        fprintf(stderr, "24 %% %d = %d\n", s[i], 24%s[i]);
    }
    return 0;
}
```
then it prints the expected results:
```
(python2) C:\Users\HPPC\mila\dev\cpp>g++ test.cpp -o test

(python2) C:\Users\HPPC\mila\dev\cpp>test
24 / 2 = 12
24 % 2 = 0
24 / 10 = 2
24 % 10 = 4
```

So, this is a very strange problem. From now I can't determine if it comes from my specific environment.

Maybe it could come from my cuDNN version (5110), but now I don't find anymore how to download version 5105 from NVIDIA web site.

@abergeron @nouiz @lamblin 


Discussion:
`s[i]` is a size_t (64 bits) and %d prints 32 bits.  Your original `s[i]` is probably bigger than 4G.
Ok ! So I have rewritten the frprintf:

```
    fprintf(stderr, "GPU 24/%Iu == %Iu\n", s[i], 24/s[i]);
    fprintf(stderr, "GPU 24%%%Iu == %Iu\n", s[i], 24%s[i]);
    fprintf(stderr, "GPU 24+%Iu == %Iu\n", s[i], 24+s[i]);
    fprintf(stderr, "GPU 24-%Iu == %Iu\n", s[i], 24-s[i]);
    fprintf(stderr, "GPU 24*%Iu == %Iu\n", s[i], 24*s[i]);
```

And now it prints:
```
GPU 24/8589934594 == 0
GPU 24%8589934594 == 24
GPU 24+8589934594 == 8589934618
GPU 24-8589934594 == 18446744065119617046
GPU 24*8589934594 == 206158430256
```

that is too big. I continue investigating. At some points these arrays (w, s, p) receive some npy_intp values, maybe the problem comes from here.
npy_intp is signed, whereas size_t is unsigned so that may be the problem.  You should use ssize_t for strides (since they can be negative).
The issue seems to be that the C code assumes that the numpy arrays have the dtype `intp`, but does not check it at all.
On 64-bit windows, intp is 64-bit, but the ints passed would probably be 32-bit. However, it would also be possible to pass explicitly arrays of lower precision to the `make_node`, which would trigger the issue.
The easiest fix would probably be to force the use of explicit 64-bit as inputs of the node, by calling `.astype('int64')` on the right variables in the `make_node`.
@lamblin Indeed ! At some points (e.g. https://github.com/Theano/Theano/blob/master/theano/gpuarray/pool.c#L244 ), when I replaced `(npy_intp*)` with `(DTYPE_INPUT_1*)`, `(DTYPE_INPUT_2*)`, etc. ... then it partly resolved the problem. And indeed the integers passed are `int32` on my computer.

I will try the fix you suggest, as it could avoid to modify too much lines of C code (there are many assumed castings to `npy_intp*`, not only in `pool.c`, but also in `pool_max_grad.c` for example).

Resolving patch:
diff --git a/theano/gpuarray/pool.py b/theano/gpuarray/pool.py
--- a/theano/gpuarray/pool.py
+++ b/theano/gpuarray/pool.py
@@ -68,6 +68,10 @@ def make_node(self, inp, ws, stride=None, pad=None):
         if pad.dtype not in theano.tensor.int_dtypes:
             raise TypeError('Padding parameters must be ints.')
 
+        ws = theano.tensor.cast(ws, 'int64')
+        stride = theano.tensor.cast(stride, 'int64')
+        pad = theano.tensor.cast(pad, 'int64')
+
         return Apply(self, [inp, ws, stride, pad], [inp.type()])
 
     def get_params(self, node):
@@ -183,6 +187,11 @@ def make_node(self, inp, out, out_grad, ws, stride=None, pad=None):
             raise TypeError('Stride parameters must be ints.')
         if pad.dtype not in theano.tensor.int_dtypes:
             raise TypeError('Padding parameters must be ints.')
+
+        ws = theano.tensor.cast(ws, 'int64')
+        stride = theano.tensor.cast(stride, 'int64')
+        pad = theano.tensor.cast(pad, 'int64')
+
         return Apply(self, [inp, out, out_grad, ws, stride, pad], [inp.type()])
 
     def get_params(self, node):
@@ -257,6 +266,11 @@ def make_node(self, inp, out_grad, ws, stride=None, pad=None):
             raise TypeError('Stride parameters must be ints.')
         if pad.dtype not in theano.tensor.int_dtypes:
             raise TypeError('Padding parameters must be ints.')
+
+        ws = theano.tensor.cast(ws, 'int64')
+        stride = theano.tensor.cast(stride, 'int64')
+        pad = theano.tensor.cast(pad, 'int64')
+
         return Apply(self, [inp, out_grad, ws, stride, pad], [inp.type()])
 
     def get_params(self, node):
@@ -334,6 +348,11 @@ def make_node(self, inp, out, out_grad, ws, stride=None, pad=None):
             raise TypeError('Stride parameters must be ints.')
         if pad.dtype not in theano.tensor.int_dtypes:
             raise TypeError('Padding parameters must be ints.')
+
+        ws = theano.tensor.cast(ws, 'int64')
+        stride = theano.tensor.cast(stride, 'int64')
+        pad = theano.tensor.cast(pad, 'int64')
+
         return Apply(self, [inp, out, out_grad, ws, stride, pad], [inp.type()])
 
     def get_params(self, node):
@@ -402,13 +421,17 @@ def make_node(self, inp, eval_point, ws, stride=None, pad=None):
         pad = as_tensor_variable(pad)
         assert ws.ndim == stride.ndim and ws.ndim == pad.ndim
         assert ws.ndim == 1
-        if not ws.dtype.startswith('int'):
+        if ws.dtype not in theano.tensor.int_dtypes:
             raise TypeError('Window shape parameters must be ints.')
-        if not stride.dtype.startswith('int'):
+        if stride.dtype not in theano.tensor.int_dtypes:
             raise TypeError('Stride parameters must be ints.')
-        if not pad.dtype.startswith('int'):
+        if pad.dtype not in theano.tensor.int_dtypes:
             raise TypeError('Padding parameters must be ints.')
 
+        ws = theano.tensor.cast(ws, 'int64')
+        stride = theano.tensor.cast(stride, 'int64')
+        pad = theano.tensor.cast(pad, 'int64')
+
         return Apply(self, [inp, eval_point, ws, stride, pad], [eval_point.type()])
 
     def get_params(self, node):


Test patch: