When we attempt to cache an array with more than 200 items this will estimate the size by sampling 100 items from the array. There's no modifier to the final size estimate so if we miss a few really large items we're gonna underestimate the amount of memory that we need and that we will use. Spark uses this estimate to determine how much memory it has left so this def feels like a possible culprit to the cache issues. If we fill up more than we have spark will think that there's lots of memory left when there isn't and will attempt to store more in memory then it can fit. This hack just removes the sampling to make the point clear.