Reduce number of non-zero similarities by only computing
those pairs that are neighbors or share at least one neighbor.