Attempt to optimize TravisCI build time.

Most time is spent building dependencies. Reducing the number that
need to be installed seems like the best way to do this, without
compromising on which tests can be run.