Allow VG_INVALID_HANDLE argument to vgSetPaint().
Set error on bad paintMode to vgSetPaint().