Is it really that much of a performance win for this not to be `Option<T>`? (so you can transfer ownership temporarily when growing.