Using the shared_ptr to vector was causing some funkiness but i realize with move constructors we don't need to worry about an expensive vector copy (it should use the move constructor automatically i think?)