Don't cache small collections

I have noticed when using two systems my TV shows get messed up if I
only have one in a folder.  No need to cache small dirs anyway.  This
should help with watched status stuff for TV too.