Hopefully fixed this.serverRemoved issue

When two players were playing simultaneously, things got out of sync. My
initial thoughts are that this serverRemoved variable was getting
switched half way through being used. Hopefully this will fix that.