Minor optimizations to neighbor list construction