[Fix] Fixed some bugs in Nelder-Mead optimization and BFGS minimization.

If the function value at each step in the process is NaN or positive/negative infinity, the process is stopped.