Fix precision issues with new conversion shader

Turns out that on some adapters, due to some sort of internal GPU
precision error, fmod(x, y) can return x when x == y, wich is incorrect
(and no, they were actually equal, not off due to precision errors).

This would cause the shader to sample wrong coordinates on the edges
sometimes.  Just adding 0.1 to the x value before being put in to fmod
and then flooring the result after fixes the issue.