> How would the "count of all unique terms across all documents" makes sense for a single SortedSetDocValues instance. 

Because a SortedSetDocValues loads all info from the index segment.
Also it's definitely not the average as in all single-field tests I've been debugging, it always runs out after the first (as expected) but allocates arrays of size 4.