Turns out OpenGL does the perspective divide for you inbetween the vertex and pixel shader. Not sure why that causes problems but it does...