Fix BigDecimal rounding

BigDecimal should not be to_int'ing when a scale is given.
Proper BigDecimal#mode error for bad args

Raise ArguementError (not TypeError) for non-boolean second argument for
BigDecimal#mode.
Merge pull request #1683 from dmarcotte/test-fix-bigdec

Fix BigDecimal tests on master