Swap references to rows rather than each individual fmpz