ENH: Remove redundant calculations of computing residual