Optimization tool for finding orthogonal matrix