Use least squares to estimate rotation matrix based on observations