libretro/mupen64plus-libretro — pull request #162 by cxd4 (2014-12-27 18:01:23+00:00)
Title: `blitter_buf' needs to allocate space based on VI_V_SYNC_REG limit.

The allocation formula for the `PreScale` or `blitter_buf` buffers is incorrectly defined based on angrylion's adjustable `screen_width` and `screen_height` constants. Those values were implemented by angrylion as a quick means of changing the window size to which DirectDraw scaled the blit (something I imagine the RetroArch frontend is already capable of handling externally). The RCP's VI, on the other hand, needs to write memory to `PreScale` based on the VI_V_SYNC_REG limit, even though only the first 480 lines are viewable in NTSC display mode.

Fundamentally, the difference in this formula is that `screen_height = SCREEN_HEIGHT` is only 480, while `PRESCALE_HEIGHT` is angrylion's calculated limit for all practical values of VI_V_SYNC_REG: 625, the traditional value for PAL (with the NTSC limit being 525 usually). Since the old formula allocated only 640 * 480 * 4 bytes instead of 640 * 625 * 4 bytes, certain ROMs--chiefly demos from the look of things (GBLator for CD64, Absolute Crap 2, etc.)--will fail under a segfault within mupen64plus-libretro before this commit.

For example, at this line of code:
https://github.com/libretro/mupen64plus-libretro/blob/master/angrylionrdp/n64video_vi.c#L220

During the `while (i < vactivelines)` loop, `i` can eventually encounter a value of 481, as `vactivelines` can be greater than 480 so long as it does not exceed VI_V_SYNC_REG's key value. When this happens, the zerobuf function will eventually attempt to `memset(&PreScale[481*PRESCALE_WIDTH + h_start], 4 * hres);`, which is guaranteed to be out-of-bounds and probably raise an access violation.