Don't use copy-and-swap here, since it requires 3 calls to shared_ptr move assignment instead of just 1