This relates to my earlier question - do we even want a non-resumable benchmark? I would argue that we do not. Hence the API distinction between buildPlannerBenchmark() and buildResumablePlannerBenchmark() should, in my opinion, just go away.