Do not rebuild the neighbor list every time step