Add -fno-spec-constr when building on Travis CI.

It should save a lot of memory, and hopefully make the builds
succeed more often.