Don't swap buffers unless we have redrawn