Check overlap with all positive and negative samples