reinsert doesPointTo in swap/move

Basically reverts #1390 now that we have a more usable `doesPointTo`.

Pinging @DmitryOlshansky and @andralex.

I removed a unittest that was just kind of wrong (swapping with internal pointers), which I had introduced, but I'm not sure why...