Revert "Speed up Travis builds by using the cache"

This reverts commits 74092eae06afa2065fb74793aaa402c313d11de1
and e623688d40995ba4423872152be1fce33b865af7.