Well that was dumb. Revert "Skip more gems for Travis"

This reverts commit 04b9f331ca3dfe18b10b5b9e4344f93d2c13aaaf.