Server maintains correct list of clients which may display opponents accurately