Remove colors only where we need to remove them