Thanks for the explanation. In that case, I would do this explicitly using a mapPartitions. Otherwise it is too convoluted / confusing.