Repository: Theano/Theano

Issue:
gradient of binary_crossentropy(sigmoid(), ...) doesn't correctly optimize if reshape before or after sigmoid
``` python
import theano
import theano.tensor as T
from theano.tensor.nnet import binary_crossentropy, sigmoid
import numpy
a = theano.tensor.tensor4()

In [27]: T.grad(binary_crossentropy(sigmoid(a.reshape((-1, 1))), 1).sum(), a).eval({a: numpy.array([[[[30.]]]])})
Out[27]: array([[[[ nan]]]])

In [28]: T.grad(binary_crossentropy(sigmoid(a).reshape((-1, 1)), 1).sum(), a).eval({a: numpy.array([[[[30.]]]])})
Out[28]: array([[[[ nan]]]])

In [29]: T.grad(binary_crossentropy(sigmoid(a), 1).sum(), a).eval({a: numpy.array([[[[30.]]]])})
Out[29]: array([[[[-0.]]]])
```



Discussion:
What's really puzzling is that it cares if there's a reshape inside the sigmoid. What the hell.

@mgermain @abergeron Was just told by Adriana that you guys were looking for a similar bug. Looks like this might be the same...

Note that in this particular case I can work around it with

`T.grad(binary_crossentropy(sigmoid(a.flatten(2)), 1).sum(), a).eval({a: numpy.array([[[[30.]]]])})`

So why does a reshape argument to sigmoid trigger the bug?! I thought we were past the era where we couldn't depend on Theano stability optimizations working reliably.

For all optimization, if you introduce another op in the middle of the
source graph, then we must update the optimization to understand it. I
don't know how to do this automatically for all optimization for a given
extra op. If you know such a way, tell me! We already generalized many
stability optimization to help on this.

In this case, it is the optimization local_greedy_distributor that isn't
being applied in this case. It don't remove the division. This cause a
division by 0.

We have spent more time on the categorical crossentropy then the binary
crossentropy. This could be another work around.

On Wed, Feb 17, 2016 at 6:41 PM, David Warde-Farley <
notifications@github.com> wrote:

> Note that in this particular case I can work around it with
> 
> T.grad(binary_crossentropy(sigmoid(a.flatten(2)), 1).sum(), a).eval({a:
> numpy.array([[[[30.]]]])})
> 
> So why does a reshape argument to sigmoid trigger the bug?! I thought we
> were past the era where we couldn't depend on Theano stability
> optimizations working reliably.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/Theano/Theano/issues/4086#issuecomment-185461042.

@nouiz The "op in the middle" explains why the `sigmoid(a).reshape((-1, 1))` graph fails to optimize. It does not explain this:

``` python
In [6]: grad(binary_crossentropy(sigmoid(a.flatten(2)), 1).sum(), a).eval({a: numpy.array([[[[30.]]]])})
Out[6]: array([[[[-0.]]]])

In [7]: grad(binary_crossentropy(sigmoid(a.reshape((-1, 1))), 1).sum(), a).eval({a: numpy.array([[[[30.]]]])})
Out[7]: array([[[[ nan]]]])
```

These both have an op inside the sigmoid() call. This shouldn't affect the pattern substitution outside the sigmoid. But somehow it does.

Well, the log(sigmoid()) in the forward graph seems to be correctly optimized in both cases, but the simplification of the division induced by the gradient of log() seems to be more difficult in the second time, I'm not sure why yet.

Here is the graph of the last case (without elemwise fusion, as I think it is clearer that way):

```
Reshape{4} [id A] <TensorType(float64, 4D)> ''   15
 |Elemwise{Mul}[(0, 0)] [id B] <TensorType(float64, col)> ''   14
 | |Elemwise{Mul}[(0, 0)] [id C] <TensorType(float64, col)> ''   13
 | | |Elemwise{Add}[(0, 0)] [id D] <TensorType(float64, col)> ''   12
 | | | |Elemwise{true_div} [id E] <TensorType(float64, col)> ''   10
 | | | | |TensorConstant{(1, 1) of -1.0} [id F] <TensorType(float64, (True, True))>
 | | | | |InplaceDimShuffle{0,x} [id G] <TensorType(float64, col)> ''   8
 | | | |   |Elemwise{scalar_sigmoid} [id H] <TensorType(float64, col)> ''   6
 | | | |     |Reshape{2} [id I] <TensorType(float64, col)> ''   4
 | | | |       |<TensorType(float64, 4D)> [id J] <TensorType(float64, 4D)>
 | | | |       |TensorConstant{[-1  1]} [id K] <TensorType(int64, vector)>
 | | | |Elemwise{true_div,no_inplace} [id L] <TensorType(float64, col)> ''   11
 | | |   |TensorConstant{(1, 1) of -0.0} [id M] <TensorType(float64, (True, True))>
 | | |   |InplaceDimShuffle{0,x} [id N] <TensorType(float64, col)> ''   9
 | | |     |Elemwise{sub} [id O] <TensorType(float64, col)> ''   7
 | | |       |TensorConstant{(1, 1) of 1.0} [id P] <TensorType(float64, (True, True))>
 | | |       |Elemwise{scalar_sigmoid} [id H] <TensorType(float64, col)> ''   6
 | | |InplaceDimShuffle{0,x} [id G] <TensorType(float64, col)> ''   8
 | |InplaceDimShuffle{0,x} [id N] <TensorType(float64, col)> ''   9
 |MakeVector{dtype='int64'} [id Q] <TensorType(int64, vector)> ''   5
   |Shape_i{0} [id R] <TensorType(int64, scalar)> ''   3
   | |<TensorType(float64, 4D)> [id J] <TensorType(float64, 4D)>
   |Shape_i{1} [id S] <TensorType(int64, scalar)> ''   2
   | |<TensorType(float64, 4D)> [id J] <TensorType(float64, 4D)>
   |Shape_i{2} [id T] <TensorType(int64, scalar)> ''   1
   | |<TensorType(float64, 4D)> [id J] <TensorType(float64, 4D)>
   |Shape_i{3} [id U] <TensorType(int64, scalar)> ''   0
     |<TensorType(float64, 4D)> [id J] <TensorType(float64, 4D)>
```

Two optimizations seem to be missing:
- removing the useless `dimshuffle{0, x}` on a column, this should be taken care of by #4071 or a follow-up, and I don't think it has an impact in this case
- Simplifying `[id L]`, or `0 / (1 - sigmoid(...))` into 0, which would remove the useless addition (`[id D]`) and make it possible to simplify the ratio by `[id G]` (or `[id H]`, `sigmoid(...)`).

I guess that the second optimization (`0 / x => 0`) exists, but for some reason is not registered that late?

The last optimization don't exist:

> > > import theano
> > > v=theano.tensor.vector()
> > > f=theano.function([v], 0./v)
> > > theano.printing.debugprint(f)
> > > Elemwise{true_div,no_inplace} [id A] ''   0
> > >  |TensorConstant{(1,) of 0.0} [id B]
> > >  |<TensorType(float64, vector)> [id C]

Should we add it? We stoped adding opt that will remove nan in the graph.
If v contain 0s or nan, then we remove nan. Removing NaN that should be
there sometimes isn't always good.

But as currently, we already do this in many cases when we should not and
that this new opt do the same but also remove nan that should not exit, I
think we should add it. But I would like the agrement of multiple people on
this. Also, I would like to tag this opt as remove_nan or something, so
allow more easily people to disable the opt that hide nan.

Another option would be to make an LogSigmoid op like LogSoftmaxand get it
used (so move the call to softmax inside binary_crossentropy. Does this
make sense? A quick chat with @carriepl seem to indicate that this would
help. But this is more work then adding the other optimization.

On Thu, Feb 18, 2016 at 10:48 PM, Pascal Lamblin notifications@github.com
wrote:

> Here is the graph of the last case (without elemwise fusion, as I think it
> is clearer that way):
> 
> Reshape{4} [id A] <TensorType(float64, 4D)> ''   15
>  |Elemwise{Mul}[(0, 0)] [id B] <TensorType(float64, col)> ''   14
>  | |Elemwise{Mul}[(0, 0)] [id C] <TensorType(float64, col)> ''   13
>  | | |Elemwise{Add}[(0, 0)] [id D] <TensorType(float64, col)> ''   12
>  | | | |Elemwise{true_div} [id E] <TensorType(float64, col)> ''   10
>  | | | | |TensorConstant{(1, 1) of -1.0} [id F] <TensorType(float64, (True, True))>
>  | | | | |InplaceDimShuffle{0,x} [id G] <TensorType(float64, col)> ''   8
>  | | | |   |Elemwise{scalar_sigmoid} [id H] <TensorType(float64, col)> ''   6
>  | | | |     |Reshape{2} [id I] <TensorType(float64, col)> ''   4
>  | | | |       |<TensorType(float64, 4D)> [id J] <TensorType(float64, 4D)>
>  | | | |       |TensorConstant{[-1  1]} [id K] <TensorType(int64, vector)>
>  | | | |Elemwise{true_div,no_inplace} [id L] <TensorType(float64, col)> ''   11
>  | | |   |TensorConstant{(1, 1) of -0.0} [id M] <TensorType(float64, (True, True))>
>  | | |   |InplaceDimShuffle{0,x} [id N] <TensorType(float64, col)> ''   9
>  | | |     |Elemwise{sub} [id O] <TensorType(float64, col)> ''   7
>  | | |       |TensorConstant{(1, 1) of 1.0} [id P] <TensorType(float64, (True, True))>
>  | | |       |Elemwise{scalar_sigmoid} [id H] <TensorType(float64, col)> ''   6
>  | | |InplaceDimShuffle{0,x} [id G] <TensorType(float64, col)> ''   8
>  | |InplaceDimShuffle{0,x} [id N] <TensorType(float64, col)> ''   9
>  |MakeVector{dtype='int64'} [id Q] <TensorType(int64, vector)> ''   5
>    |Shape_i{0} [id R] <TensorType(int64, scalar)> ''   3
>    | |<TensorType(float64, 4D)> [id J] <TensorType(float64, 4D)>
>    |Shape_i{1} [id S] <TensorType(int64, scalar)> ''   2
>    | |<TensorType(float64, 4D)> [id J] <TensorType(float64, 4D)>
>    |Shape_i{2} [id T] <TensorType(int64, scalar)> ''   1
>    | |<TensorType(float64, 4D)> [id J] <TensorType(float64, 4D)>
>    |Shape_i{3} [id U] <TensorType(int64, scalar)> ''   0
>      |<TensorType(float64, 4D)> [id J] <TensorType(float64, 4D)>
> 
> Two optimizations seem to be missing:
> - removing the useless dimshuffle{0, x} on a column, this should be
>   taken care of by #4071 https://github.com/Theano/Theano/issues/4071
>   or a follow-up, and I don't think it has an impact in this case
> - Simplifying [id L], or 0 / (1 - sigmoid(...)) into 0, which would
>   remove the useless addition ([id D]) and make it possible to simplify
>   the ratio by [id G](or [id H], sigmoid%28...%29).
> 
> I guess that the second optimization (0 / x => 0) exists, but for some
> reason is not registered that late?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/Theano/Theano/issues/4086#issuecomment-186038767.


Resolving patch:
diff --git a/theano/tensor/opt.py b/theano/tensor/opt.py
--- a/theano/tensor/opt.py
+++ b/theano/tensor/opt.py
@@ -5270,6 +5270,7 @@ def local_intdiv_by_one(node):
 
 
 @register_canonicalize
+@register_specialize
 @gof.local_optimizer([T.int_div, T.true_div])
 def local_zero_div(node):
     """0 / x -> 0


Test patch:
diff --git a/theano/tensor/nnet/tests/test_nnet.py b/theano/tensor/nnet/tests/test_nnet.py
--- a/theano/tensor/nnet/tests/test_nnet.py
+++ b/theano/tensor/nnet/tests/test_nnet.py
@@ -30,7 +30,8 @@
                                 Prepend_scalar_to_each_row,
                                 relu,
                                 h_softmax,
-                                elu)
+                                elu,
+                                binary_crossentropy)
 from theano.tensor import matrix, vector, lvector, scalar
 
 
@@ -1684,3 +1685,17 @@ def test_elu():
     for alpha in 1.5, 2, -1, -1.5, -2:
         y = elu(x, alpha).eval({x: X})
         utt.assert_allclose(y, numpy.where(X > 0, X, alpha * (numpy.exp(X) - 1)))
+
+
+def test_binary_crossentropy_reshape():
+    # Reported as https://github.com/Theano/Theano/issues/4086
+    a = tensor.tensor4('a')
+    for c in (binary_crossentropy(sigmoid(a.reshape((-1, 1))), 1).sum(),
+              binary_crossentropy(sigmoid(a).reshape((-1, 1)), 1).sum()):
+
+        ga = theano.grad(c, a)
+        # This only works when "specialize" options are included
+        mode = theano.compile.get_default_mode().including('fast_run')
+        fga = theano.function([a], ga, mode=mode)
+        utt.assert_allclose(fga(numpy.array([[[[30.]]]], dtype=config.floatX)),
+                            numpy.zeros((1, 1, 1, 1), dtype=config.floatX))