Remove specific cache size test.

We're already testing the ability to override the cache size and testing
that the cache is enabled by default, so this doesn't provide a lot of
extra coverage and breaks when we change the default value.