BVLC/caffe — pull request #209 by jeffdonahue (2014-03-13 21:10:03+00:00)
Title: Compute loss in the forward pass

(PR currently empty but opening for discussion as per the dev guidelines.)

Caffe currently computes loss in the backward pass. From looking at loss layer code and talking to @Yangqing, this was done to save a bit of time in the loss layer as intermediate computations are often shared between the gradient and loss computations. However, this could be remedied (at the cost of a bit of memory - probably pretty insignificant in the loss layers) by storing as needed these intermediate computations in the forward pass and reusing them in the backward pass.

The main motivating factors are:

1) Sometimes it may be useful to be able to compute the loss at training time without actually doing an entire backwards pass (e.g., if one wanted to perform backtracking line search to set the step size/learning rate).

2) The forward pass seems to me the natural place to perform loss computation as it is where the "inference" is supposed to happen.

I think the only cost to this change would be the aforementioned bit of memory in the loss layers, but feel free to discuss - maybe there are issues I've overlooked.