Cache each virtualenv's site-packages folder, instead of the whole ~/virtualenv

Travis inits every python version's virtualenv before starting the test, so we
would cache them all.