BUG: inf turned to nan when calling np.triu/np.tril, fixes #4859

Changes the current multiplication method to zero the items to remove
with boolean indexing.