Remove our pieces from safe checks if it's our turn