BACKPORT: Fix inconsistent behavior in diffsets on remove/add combination.

This is a backport of 840cd335fe08642b38b622b73d984c9096cecf4c from 2.1 to 2.0, since 2.0 exhibits the same problems.