Only poll for changes if it's not our turn