Don’t die when connecting to a full server.

Fixes an oversight in #6573.
This was to prevent players joining as a spectator with an invalid color, and then keeping that color if they subsequently join a slot.