Add port number to set_neighbors