Longs are not guaranteed to be the same size as pointers like I thought.
Switch to just doing pointer arithmetic, and using size_t when we have to