Not sure why... but it seems like uncommenting  vi.bits_per_pixel = PIXEL_SIZE * 8; in graphics.c fixed the major screen lag I was seeing.