Remove per-frame normalize, do it once when setting the up vector