Only trim the cache if something has been added since the last trim.