Yes, you can use Masking to apply the swap only to the colors that you want to swap.