Properly check for not being your turn