Alt+Shift+K can be used to remove some region from the linked selection.