Remove interfaces that have been unselected from CP config properly