If a 3D driver just left the chip set up for 3D drawing, then when X tried to draw something on the screen, it would wind up on the back buffer, possibly depth buffered and blended, which is not what X wants. This is wasteful of command bandwidth. In an ideal world, X would generate DMA command buffers that could be nicely scheduled between 3D DMA command buffers.
X can draw.
This does not seem to be a huge overhead, but it is unfortunate in principle. The registers that control the mouse cursor are left unprotected, so moving the mouse cursor does not cause signals to be fired. Thankfully, matrox has these on a separate 4k page from the drawing registers, and they don't interfere with each other. Whenever X tries to draw something to the screen when 3D rendering is happening, it will hit the signal handler and basically spin in place until all the 3D drawing completes. This ruins asynchronous performance, and in the worst case, can cut 3D framerates in half if the load was evenly balanced between the CPU and the graphics accelerator.