Remove our pieces from safe checks, not including pawns