fix sign and division errors in least squares