@dbtsai This test still looks strange, because both gradient descent and L-BFGS are monotonic if the line search is correctly implemented. In our gradient descent implementation, step sizes are pre-defined, so it may not be monotonic if the initial step size is too large. However, L-BFGS uses line search, so maybe you should test the sequence of loss is strictly decreasing instead of 80 percent of them.