I would create a separate QuerySegmentWalker so the merged results caching stays independent of the compute results caching.
CachingClusteredClient already does a lot of things, no need to make it more complicated