We decided to look into OwnedPointerVector for this.