BUG: gh-3300, Fix missing return value in test_rational.c.src.

Some error returns were returning nothing rather than NULL.

Closes #3300.