Forgot you cannot remove all NICs from a team, so add first and remove after.
Using Foreach didn't work like expected.