Simplify symmetric/sparse loop decisions

Simplify the tests for similarity sparsity and symmetry when choosing the
neighbors to consider.