BVLC/caffe — pull request #981 by jeffdonahue (2014-08-25 23:29:21+00:00)
Title: Make eltwise product gradient stabler (fixes issues with WITHIN_CHANNEL LRN in cifar_full example)

The eltwise product layer's forward computation is (given inputs x, y, z) `p := x .* y .* z`. Previously I was computing the gradient w.r.t. `x` as `p ./ x` (analogously for `y`, `z`); this changes the layer to by default compute the gradient w.r.t. `x` as `y .* z`, which is asymptotically slower in the number of inputs (O(n^2) instead of O(n)), but stabler than dividing by the potentially near-zero `x`. For the case of two inputs (which is probably 99% of the uses of this layer, including the CIFAR example) it's actually faster (just copy the other input) and more accurate, but if you have lots of inputs and you're sure dividing by them will not cause any instability (e.g. if you specifically took measures to condition the inputs as such) you can still set the `stable_prod_grad: false` option for the old method.

This division by near-zero was causing NaNs in the cifar_full example, which uses the eltwise product as part of the WITHIN_CHANNEL LRN computation.