Removed rubys that don't have require_relative from build.

The gem should still work but I don't use those versions so I can't
support them.  This keeps our build nicer and doesn't bog down
Travis-CI.