Only remove from conn_tree when the port-tree is empty. Duh.