improve factorial, closes #6579

add overflow and domain error (negative) checking, improve type-stability,
and make it faster using table lookup