Tracking dirty visibility peers in order to only show visible peers