mruby-math: define and use `Math::DomainError`

Domain errors are detected as per ISO C99 and may differ from CRuby.
Merge pull request #2177 from cremno/math-domain-error

mruby-math: define and use `Math::DomainError`