Cleanup refresh to make the cleanup optional and avoid taking the lock
if we don't delete anything.