NLG Evaluation Results
Summary Assessment : Table 1 analyzes the effectiveness of comparative assessment on SummEval , where the following observations can be made:

(1) Moderate-sized LLMs are ineffective in the prompt-scoring set-up, with the best system (FlanT5-3B) achieving Spearman correlations of 10-20. The performance difference with ChatGPT prompt-scoring implies that scoring is likely an emergent ability only effective for larger LLMs.

(2) G-Eval, which uses task specific detailed prompts and continuous scores, yields significant improvements over prompt-scoring. Nonetheless, comparative assessment remains more effective than G-Eval in the majority of settings.

(3) LLMs are able to achieve considerably higher correlations in the comparative assessment set-up, with performance higher for nearly all systems. Further, comparative assessment leads to more robust performance, with most 3B+ models achieving correlations within the range of 30-50.

(4) Comparative assessment enables LLMs of under 1B to perform well, with FlanT5-770M achieving moderate correlations. However, performance improves significantly when using 3B+ LLMs, although for SummEval there are diminishing (if any) performance gains by scaling up.

(5) The best comparative assessment LLM (FlanT5-3B) is competitive with all other zero-shot methods, including ChatGPT scoring (an LLM with two orders of magnitude more parameters), and achieves the best correlation in 3 of the 4 aspects.

(6) Comparative assessment achieves competitive performance with UniEval. Although UniEval has better overall performance, UniEval was designed for bespoke tasks and aspects (it is fine-tuned on synthetic data created for particular attributes) where the results in Tables 2 and 4 show that UniEval has noticeable degradation in out-of-domain settings. In contrast, comparative assessment is zero-shot and general.

Podcast Assessment : When considering podcast summarization with long inputs of over 5k tokens on average, only Llama2 models (which have a limit of 4k tokens) were used (as FlanT5 has a limit of 1k tokens). Table 2 shows that comparative assessment yields highly impressive performance for long-spoken summarization, with comparative assessment out-competing all other baselines. Further, although prompt-scoring has good system-level correlations, the lack of granularity leads to poor summary-level performance.

Dialogue Assessment : Next, we analyze comparative assessment on TopicalChat, for evaluating conversational responses. Table 3 shows similar findings for TopicalChat as to those in SummEval, where comparative assessment again outperforms the correlations seen from prompt-scoring.

Data-to-Text Assessment : For data-to-text generation, the context is highly abstract and is a list of triples in the form of (object, relation, subject). This makes assessing the semantics challenging, as the LLM needs to parse and understand semantic triples. Table 4 shows that understanding triples is an emergent ability of LLMs, where for grammar and fluency the correlations are quite similar between the 3B and 11B/13B systems, however for semantic understanding, the 10B+ systems highly outcompete the 3B+ systems. Note that when evaluating UniEval, we used the closest attribute that they designed for, which was naturalness for both.