Fix shady pointer arithmetic, should fix the build with non-GCC compilers.