Keep track of the nodes we recently connected to