Switch to pointer representation internally

- Avoids unnecessary copying for sort.
- makes code internally consistent