Change to uint32s for offsets and sequence numbers

This makes RecordRef be 8 bytes, rather than 16, greatly reducing the
size of the refCache.
Get rid of two pointer sources in memCache

- Store timestamps as int64s
- Use a Record rather than *Record as the map value