Make Every Penny Count: Difficulty-Adaptive Self-Consistency for Cost-Efficient Reasoning
Authors: Xinglin Wang, Shaoxiong Feng, Yiwei Li, Peiwen Yuan, Boyuan Pan, Yueqi Zhang, Heda Wang, Yao Hu, Kan Li


Reviewer paper summaries: The paper "Make Every Penny Count: Difficulty-Adaptive Self-Consistency for Cost-Efficient Reasoning" introduces a novel Difficulty-Adaptive Self-Consistency (DSC) method aimed at optimizing the computational efficiency of language models in reasoning tasks. By leveraging difficulty assessments to adaptively allocate computational resources, DSC significantly reduces costs without compromising performance. It features a three-step methodology—Difficulty Ranking, Problem Partition, and Sample Size Pre-Allocation—and has been rigorously tested across arithmetic, commonsense, and symbolic reasoning tasks on six benchmarks. The results demonstrate  improvements in cost efficiency compared to existing methods, suggesting that DSC's innovative use of difficulty information can meaningfully reduce unnecessary computations, particularly for simpler tasks, thereby broadening the potential application of AI in resource-limited settings.
Summary of strengths: The focus on this paper is to reduce cost of querying GPT-4 while maintaining reasoning accuracy. The author proposes an difficulty-adaptive method to reduce sampling for reasoning, and compare their method against self-consistency, adaptive self-consistency and early-stopping self-consistency. The evaluation results show that it does reduce the cost while keeping reasoning accuracy.
Summary of weaknesses: The difficulty ranking component of the model involves multiple iterations of batch splitting and ranking, which may not accurately represent the true difficulty of a problem. The resultant difficulty ranking is relative, depending heavily on the comparative difficulty of other problems within the same batch. While the underlying concept is straightforward, it exhibits limited novelty, suggesting potential areas for further innovation and refinement.
Comments, suggestions and typos: The author should mention the full name of adaptive self-consistency and early-stopping self-consistency for the abbreviation.
Overall assessment scores: 2