Fix some overflows by R_ScaleFromGlobalAngle()

Setting max in this function to a higher value than 64 improves the
"wobbly" vertexes that can occur, but this causes overflows to occur.
Now that I've actually found an instance of this occurring (DOOM2.WAD,
MAP06: The Crusher, angle = 0, x,y,z = (511, 797, -512)), I've lowered
the max from 1024 to 512. I guess overflows will still occur, but until
I find another one, this can stay.