Return a BigDecimal::NAN if arg is (Float|BigDecimal)::NAN

* needed separate treatment for Float and BigDecimal NAN
* test for float NAN was not present in MRI implementation
Return Math::DomainError if arg zero or negative