Repository: Theano/Theano

Issue:
tensor.abs_() returning negative values in Theano 1.0.0
When I upgraded from Theano 0.9.0 to 1.0.0 my code just broke. I narrowed down the issue to this minimal working example:

```
import theano
import theano.tensor as T
import numpy as np

x = T.dmatrix('x')

f = theano.function([x], T.mean(T.abs_(x), axis=-1))

a = np.array([[-1.], [0.], [1.]])

print f(a)
```

In Theano 0.9.0, the result is: [1.  0.  1.]
In Theano 1.0.0, the result is: [-1.  0.  1.]

Curiously, Theano 1.0.0 gives [-1. 0. 1.] on the GPU, but [1.  0.  1.] on the CPU. Could this be an issue with libgpuarray?



Discussion:
I confirm the problem (tested on Windows), quite strange.
About this issue, the problem is introduced by my commit c2c2879365b14c259a9e69a22cf49e7ddc2d40fe, where sum of abs is optimized with GpuDnnReduction.

In the code above, the tensor to reduce has shape `(3, 1)` (`a = np.array([[-1.], [0.], [1.]])`), and the axis to reduce is the last one, which has size `1`, then the code goes into this section: https://github.com/Theano/Theano/blob/master/theano/gpuarray/c_code/dnn_redux.c#L89 . Nothing is computed, the output returned is just a copy of the input.

It seems that this is to avoid passing an output which will have the same shape as the input (here `(3, 1)`) to the cuDNN reductiion function, as it does not like it (`CUDNN_STATUS_BAD_PARAM`).

@abergeron @lamblin @nouiz 
Can you comment that bypass for that case and see if cudnn does the right thing?

In the mean time, you can use this theano flag as a workaround: optimizer_excluding=local_gpu_elemwise_careduce
If I comment (`/* || rsz == 1 */`), I got the `CUDNN_STATUS_BAD_PARAM`.

When using `optimizer_excluding=local_gpu_elemwise_careduce` it works, because the merging (abs + max -> norm1 (absmax)) is not done, so that the abs is computed by a GpuElemwise.
OK, then I think we would need a special case to handle the pre-scalar op when the output shape is the same as the input.
And we should report that to NVIDIA. I'll do that.

Resolving patch:
diff --git a/theano/gpuarray/opt.py b/theano/gpuarray/opt.py
--- a/theano/gpuarray/opt.py
+++ b/theano/gpuarray/opt.py
@@ -1258,9 +1258,8 @@ def local_gpua_careduce(op, context_name, inputs, outputs):
             greduce = op2(
                 op.scalar_op,
                 axis=new_axis, reduce_mask=new_mask,
-                dtype=getattr(op, 'dtype', outputs[0].dtype),
-                acc_dtype=getattr(op, 'acc_dtype', None))
-
+                dtype=odtype,
+                acc_dtype=adtype)
             with inherit_stack_trace(outputs):
                 reshaped_x = x.reshape(tensor.stack(new_in_shp))
                 gpu_reshaped_x = as_gpuarray_variable(reshaped_x, context_name)


Test patch:
diff --git a/theano/gpuarray/tests/test_dnn.py b/theano/gpuarray/tests/test_dnn.py
--- a/theano/gpuarray/tests/test_dnn.py
+++ b/theano/gpuarray/tests/test_dnn.py
@@ -1611,6 +1611,53 @@ def test_dnn_reduction_absmax():
         utt.assert_allclose(np.max(np.abs(M_val), axis=axis), f(M_val))
 
 
+def test_dnn_reduction_axis_size_one():
+    if not dnn.dnn_available(test_ctx_name) or dnn.version(raises=False) < 6000:
+        raise SkipTest(dnn.dnn_available.msg)
+
+    for dtype in ('float16', 'float32', 'float64'):
+        for shape, axis in [[(1, 2, 3), 0],
+                            [(2, 1, 3), 1],
+                            [(2, 3, 1), 2],
+                            [(1, 5, 1), (0, 2)],
+                            [(4, 1, 6, 1), (1, 3)]]:
+
+            x = theano.tensor.TensorType(dtype=dtype, broadcastable=[False] * len(shape))()
+            sum = x.sum(axis=axis)
+            sum_squares = (x**2).sum(axis=axis)
+            sum_abs = abs(x).sum(axis=axis)
+            absmax = abs(x).max(axis=axis)
+
+            cpu_f = theano.function([x], [sum, sum_squares, sum_abs, absmax], mode=mode_without_gpu)
+            f1 = theano.function([x], sum, mode=mode_with_gpu)
+            f2 = theano.function([x], sum_squares, mode=mode_with_gpu)
+            f3 = theano.function([x], sum_abs, mode=mode_with_gpu)
+            f4 = theano.function([x], absmax, mode=mode_with_gpu)
+
+            for fn, red_op in ((f1, 'add'), (f2, 'norm2'), (f3, 'norm1'), (f4, 'absmax')):
+                assert any(isinstance(node.op, dnn.GpuDnnReduction) and node.op.red_op == red_op
+                           for node in fn.maker.fgraph.apply_nodes)
+
+            xval = np.random.uniform(-10, -1, size=shape).astype(dtype)
+            if isinstance(axis, int):
+                xval_reshaped = xval.reshape(shape[:axis] + shape[(axis + 1):])
+            else:
+                xval_reshaped = xval.reshape([n for i, n in enumerate(shape) if i not in axis])
+            test_val = abs(xval_reshaped)
+
+            val_sum, val_sum_squares, val_sum_abs, val_absmax = f1(xval), f2(xval), f3(xval), f4(xval)
+            cpu_val_sum, cpu_val_sum_squares, cpu_val_sum_abs, cpu_val_absmax = cpu_f(xval)
+
+            utt.assert_allclose(cpu_val_sum, val_sum)
+            utt.assert_allclose(cpu_val_sum_squares, val_sum_squares)
+            utt.assert_allclose(cpu_val_sum_abs, val_sum_abs)
+            utt.assert_allclose(cpu_val_absmax, val_absmax)
+            utt.assert_allclose(xval_reshaped, val_sum)
+            utt.assert_allclose(test_val**2, val_sum_squares)
+            utt.assert_allclose(test_val, val_sum_abs)
+            utt.assert_allclose(test_val, val_absmax)
+
+
 def dnn_reduction_strides(shp, shuffle, slice):
     utt.fetch_seed()
     inp = GpuArrayType('float32', (False,) * len(shp),