invalidate the cached min handle size when adding and removing the first handle