BVLC/caffe — pull request #363 by jeffdonahue (2014-04-25 09:45:00+00:00)
Title: Gradient check speedups

Two changes to the gradient checker that improve performance pretty significantly without any cost to unit tests coverage.

(1) 8901fe7 moves the analytic gradient computation outside the loop over bottom features and stores it throughout all finite differencing computation (extra storage should be pretty minimal since the unit tests use relatively tiny blobs).

(2) 061b0db adds a method `GradientCheckEltwise` that replaces `GradientCheckExhaustive` in many layers. In any layer where we have element-wise computation where each output is a function only of the input at the corresponding index (e.g., neuron layers), we only need to do finite-differencing to compute the diagonal of the jacobian and we know all other entries are zero.

In my tests of the entire unit test suite (`./build/test/test_all.testbin` on a k40) this gave a 73% speedup from 238,610 ms @ dev down to 66,322 ms.