Wikipedia has an interesting note that if you only need to perform a dot product with the gradient, which is the case for various optimization algorithms, then only a single evaluation is needed. We should look into taking advantage of this, but it will require more invasive changes to the implementations.