You are right.  Since the cost function is convex, the loss is guaranteed to be monotonic decreased with L-BFGS optimizer. (SGD doesn't guarantee this, and the loss may be fluctuating in the optimization process.) Will add the test for this property.