Remove OSX from our build matrix

The Travis OSX builders are incredibly inconsistent for how long
they take to run our test suite. It is currently ranging anywhere
from ~15 minutes to well beyond 50 minutes, at which point it
times out.
Merge pull request #1807 from dstufft/remove-osx

Remove OSX from our build matrix