Remove some redundant code in diff calculation