Don't listen for scroll while loading ancestor nodes