Simplify Travis CI build matrix config

:warning: ~~**Includes commits from #2204. Please review, merge first.**~~