fix a bug of removing temp_port_list too early