rational.c: ensure to be Rational

* rational.c (f_round_common): Rational is expected to be returned by
  Rational#*, but mathn.rb breaks that assumption.  [ruby-core:56177]
  [Bug #8687]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@42178 b2dd03c8-39d4-4d8f-98ff-823fe69b080e