Remove IPs from an endpoint when they are removed by the API.
Also, some minor code tidies.
Clean up logic when removing unused IPs.