IMHO its a little risky to cache all the iterations of this loop in terms of memory usage. The right thing to do is to probably hold references to them and unpersist at the end ?