Only remove unused card types from the board