DiffFactorizer: separate additions/removals into its own method