vector functions can control allocation

Starting the new pattern, out *func(*a, *b, *out).
Update matrix and renderer vec3 usage.
Calculate camera position based on viewing angle.