Revert "Trying out Travis caching to speed up builds"

This reverts commit 9ce055808a521ae7992900461def1214eca25dd2.