I thought about it, but wanted to keep the as much of the current architecture intact as possible. Spectators are non-player peers and I'm not sure if any of the rest of the Server code relies on this.