Get rid of the void pointer arithmetic here.

This broke the build on Clang and MSVC.