Only access refcount_ in non-thread safe mode

Just to be sure. Add documentation.
Speedup the other branch
Do a copy in pow.cpp rather than move

Because the refcount is not always 1 here.
Refactor the code