d3d: Improve GE debugger with render-to-tex.

Also fix a crash in oddly sized framebuffers.