BUG: gh-3300, Fix missing return value in test_rational.c.src.

Some error returns were returning nothing rather than NULL.

Closes #3300.
Merge pull request #3301 from charris/fix-test_rational

BUG: gh-3300, Fix missing return value in test_rational.c.src.