Just assign shared_ptr instead of reconstructing