Try to align the crosshair position exactly with the block boundary which is being looked at.

I'm not exactly sure why only the y-direction needs to be negative, but now it selects exactly the block you are looking at and the crosshair stays centered in the view.