Only attach players who are actually competing.
Track player in viewport, for later lookup.
Assign the correct player HUD to each player peer.