Only install things we need to test h5py