Atomic operations on shared_ptr<> is not portable

It ends up using a lock on some implementations anyway. We might need
to revisit this if it’s a bottleneck.