Only exclude rook/queen attacks