dolphin-emu/dolphin — pull request #1095 by crudelios (2014-09-16 10:18:40+00:00)
Title: Reimplement Bounding Box calculation using the software renderer.

This PR completely changes the implementation of bounding box calculation.

In the current master, bbox is calculated by finding a primitive's edges and using those values as a comparison for the bounding box calculation.

However, that code is innacurate as pixels inside the primitive may not be drawn, mainly because the old code bypassed alpha testing and z-testing (for example, if the primitive's color or texture is transparent, it is not drawn and bounding box regs should not be changed). This caused issues with the punies in Paper Mario: TTYD and also caused a hang in Super Paper Mario's world 6-1.

As far as I can tell, there is no way to find which pixels have been drawn using the hardware renderers, so I resorted to the software renderer to calculate if a pixel is going to be drawn by the hardware renderer. It works by reading the vertex buffer from VertexLoader and passing all the required vertex information (matrices, position, normals, colors and texture coordinates) to the software renderer's SetupUnit.

Since the new code relies heavily on the software renderer, bbox calculation is twice to three times as slow as the old implementation (there isn't any decrease in performance when bbox is not needed, though). However, as far as I can tell, it is nearly fully accurate (with the exception of z-testing, which cannot be trivially implemented) and it fixes all the remaining bounding box bugs.

Please note that this changes the software renderer code a little bit. I'm not entirely happy with the result, and I'm open to comments about how integration with the software renderer could be improved.