Set stencil ref based on framebuffer format

It should be the same logic as SW rasterzier