Removing all values in difference instead of number of elements in the other arrays in difference