Theano/Theano — pull request #233 by delallea (2011-11-26 03:48:49+00:00)
Title: Fixed optimization for exp(x) * sigmoid(-x)

This stability optimization didn't work for at least three reasons:
1. It used tensor.neg(x) to identify -x, while in canonicalized form -x
is written -1 * x.
2. There was a typo (calling a function sigm that didn't exist).
3. It wasn't able to parse a tree of multiplications, for instance
something like (exp(x) * 2) * sigmoid(-x).

This new version fixes all three issues. In order to solve problem #3,
it uses a different approach that consists in parsing the optimized node
into a so-called "multiplication tree".

Currently, the structure of this tree is left unchanged. It would be
possible to have this optimization also flatten the tree, which would
make the code simpler (but potentially undo work performed by some other
optimizations).

NB:
- I verified that all tests pass (except those currently broken by other recent changes)
- This fixes the numerical instability problem in the deep learning tutorials (rbm.py) recently discussed on the theano-dev mailing-list, where NaNs would appear in float32