Make refreshes more stable.

Now when the viewport is moved, items are only removed if they are
no longer visible, and only added if they are newly visible.