Clean up the mapping on node down, we keep it around across tests and we've seen proxy explosions in CI due to the manager giving the wrong answer after port reuse.