Do a more complex test to decide when to rebuild the neighbor list