EllipsoidFactor: fix a lock-up due to dilating forever