Each partition will be visited twice in a butterfly step. If the previous stage is not cached or falls out cache, the cost is huge. I'm looking at the `RangeDependency` now. Maybe it can help.

Btw, I don't quite understand what do you mean by `hold references to them`. Could you elaborate?