Fixing add_remove to not include adjacent swaps