Remove page set caching which never hits

We can worry about fixing the invalidation bug later.