Some controller niceness

Adding a new controller in-game works, removing the highest numbered
controller in-game works.

So, for example, if there are 3 people playing and anyone but player 3
disconnects, it causes issues. I don't know why yet.