Unify swapping code and declare move ctor noexcept