Fix clang compiler warning in mathlib.cpp and repair test case for #5895