mv the raw pointer {swap,replace}_ptr to std::ptr + some cleanup

I don't like the `util` module in general, and `ptr` is a much better place for these.