Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models
Authors: Sumuk Shashidhar
Abhinav Chinta
Vaibhav Sahai
Zhenhailong Wang
Heng Ji
Venue: EMNLP 2023 Findings (2023)
Keywords: large-language-model
open-source
self-refinement
ranking-metric
cost-analysis
TL;DR: Evaluation of domain-agnostic, untargeted, self-refinement on open source llms and cost-sensitive ranking of SoTA refined open source models.

Abstract: The dominance of proprietary LLMs has led to restricted access and raised information privacy concerns. The SoTA open-source alternatives are crucial for information-sensitive and high-volume applications but often lag behind in performance. To address this gap, we propose (1) A generalized variant of iterative self-critique and self-refinement devoid of external influence. (2) A novel ranking metric - Performance, Refinement, and Inference Cost Score (PeRFICS) - to find the optimal model for a given task considering refined performance and cost. Our experiments show that SoTA open source models of varying sizes from 7B - 65B, on average, improve 8.2\% from their baseline performance. Strikingly, even models with extremely small memory footprints, such as Vicuna-7B, show a 11.74\% improvement overall and up to a 25.39\% improvement in high-creativity, open ended tasks on the Vicuna benchmark. Vicuna-13B takes it a step further and outperforms ChatGPT post-refinement. This work has profound implications for resource-constrained and information-sensitive environments seeking to leverage LLMs without incurring prohibitive costs, compromising on performance and privacy. The domain-agnostic self-refinement process coupled with our novel ranking metric facilitates informed decision-making in model selection, thereby reducing costs and democratizing access to high-performing language models, as evidenced by three case studies on personal computing, gaming and enterprise solutions.

Democratizing LLMs: Self-Refined Open-Source Models
An Exploration of Cost-Performance Trade-offs in
Sumuk Shashidhar" Abhinav Chinta Vaibhav Sahai Zhenhailong Wang
Heng Ji
Department of Computer Science, University of Illinois Urbana-Champaign (sumuks2, achinta3, sahai3, wangz3, hengji jengiij@illinois edu
Abstract
How many messages are sent globally a minute? Try explain your answer. Your explanation should take the reader through your reasoning step-by-stop.
The dominance of proprietary LLMs has led to restricted access and raised information privacy concerns. Higghppeforming open-source alter- natives are crucial for information-sensiteee sensitive and high-volume applications but often lag behind in performance. To address this gap, we pro- pose (1) A untargeted variant iterative self- critique and self-refinement devoid of exter- nal influence. (2) A novel ranking metric Performance, Refinement, and Inference Cost Score (PeRFICS) to find the optimal model for a given task considering refined performance and cost. Our experiments show that SoTA open source models of varying sizes from 7B 65B, on average, improve 8.2% from their baseline performance. Strikingly, even mod- els with extremely small memory footprints, such as Vicuna-7B, show a 11.74% improve- ment overall and up 25.39% improve- ment in high-creativity, open ended tasks on the Vicuna benchmark. Vicuna-133 takes a step further and outperforms ChatGPT post- refinement. This work has profound implica- tions for resource-constrained and information- sensitive environments seeking to leverage LLMs without incurring prohibitive costs, com- promising on performance and privacy. The domain-agnostic self-refinement process cou- pled with our novel ranking metric facilitates informed decision-making model selection, thereby reducing costs and democratizing ac- cess to high-perfone sigh-performin language models, as evidenced by studies.
ChatGPT Response
Baseline Response
[..]] According to (.... there were
approsimataly billion mobil collular
According 2020. calcuiste many mossages sent globally in minute, need vide number of minutes year, which is 525, 800 (80 minutes 24 hours x 385 days).
subscriptions by the end of 2019. (-) Now. lel's sesume that each person sends receives average of two lext messages day. (..) this would result in total of 12 billion text mestages baing sent or received every day.
messages
by
To calculate the number of globally minute, the need divido cally (which numbor
15.9 trillon / 525 600 - appraxmmaaaly s0.5 billion text massages sent globaily in minute. (..)
This gives messagie every minute, (..)
o Reflined Response
(-) As December 2021 (... there were around 7.2 billion mobile collutar subscriations. Assuming that each incividual cnss or recavess an avarage ge threo text massages par we have total subscriptions. received daily
messages sont
(...] we must divide the daily amount by the number of minutes in day (1440). This ylaks an eslimatted 15,000, 000 messages
received every minute. (..)
in conclusion, (-)) approximation suggests that over 15 million text messages (--)
Figure I: ChatGPT vs GPT-4X-Alpast baseline vs GPT-4X-Alpasta refined
their own LLMs; however, the performance gap with proprietary counterparts significant. Propri- etary models, albeit powerful, come with an array of concerns including the cost associated with API and fine-tuning, privacy issues, and scalability in high volume applications.
Open Source LLMs offer a potential solution to these concerns but often fall short in performance. This has prompted the development of various tech- niques to augment the capabilities of Open Source LLMs, such as fine-tuning (Chung et al., 2022), additional pre-training (Tay et al., 2022), the incor- poration of chain-of ofaiin-o-thhught,, and self-refinement (Madaan et al., 2023; Shinn et al., 2023). While fine-tuning and additional pre-training are effective, they entail additional costs and cannot be adapted to existing models.
1 Introduction
Large Language Models (LLMs) have been experi- encing seismic growth in size and capabilities, rad- ically transforming the field of Natural Language Processing (NLP). While this growth has been impressive, the dominance of proprictary LLMs such ChatGPT remains uncontested. The Open Source community has made strides in developing
One approach that holds promise is self- refinement, which can be implemented at inference time, thus mitigating the issues associated with pre-
Equal Contribution

---

training. However, existing self-refinement tech- niques often rely on fine-grained, domain-specific feedback (Madaan et al., 2023) or incorporate knowledge from external systems, such as unit tests for code generation (Shinn et al., 2023), making them less versatile.
2 Related Works
Recent advances in the development of open source Large Language Models (LLMs) have been charac- terized by the advent of Meta Al's LlaMA models (Touvron et al., 2023). Despite their smaller sizes, such as 7B, 13B, 30B and 65B in comparison to predecessors like PaLM-540B (Chowdhery et al., 2022) and Chinchilla- 70B models (Hoffmann et al., 2022), they have exhibited exceptional capabilities. As these models edge towards achieving perfor- mance on par with large proprietary models such as OpenAl's ChatGPT and Google's s Bard, it be- gins to shape the next chapter in Al development, giving rise to hyper finetuned, private, local models. Their lower VRAM requirements, further enhanced by Weight Quantization techniques such as OPTJ (Frantar et al., 2023), have greatly improved acces- sibility and usability, particularly on less powerful consumer hardware.
In this paper, we address these challenges and make two novel contributions to significantly im- prove the capabilities of Open Source LLMs. Firstly, we introduce generalized implementation of self-refinement that independent of external in- fluence, employing domain-agnostic prompts. This offers unprecedented udaptability, enabling LLMs to efficiently perform across diverse tasks with- out external influence or task context-awareness. Secondly, we introduce PeRFICS, a novel ranking metric that enables model selection for specific tasks, considering performance task-specific benchmarks, improvement through generalized self-refinement, and memory footprint.
Finetuning techniques such as Low Rank Adap- tion (LoRA) (Hu et al., 2021) present a dichotomy. Although they effectively mitigate issues like catas- trophic forgetting (Kirkpatrick et al., 2017) and en- hance the performance of models like ChatGPT, they require high-quality data and considerable fine-tuning. This leads to a computational over- head prior to model deployment, which we aim to address our work.
Through rigorous evaluations, we demonstrate that our aogain-agnostic self-refinement technique effectively bridges the performance gap between proprietary models, specifically ChatGPT, and Open Source alternatives. Remarkably, by employ- ing just single iteration of Agnosicc Self Refinement Vicuna-13B, model that can be consumer hardware when quantized, are able outperform ChatGPT across multiple domains, such as general writing, common sense reasoning, roleplay, etc.
Inference time methods like the Chain of Thought (CoT) (Wei et al., 2023) and Self- refinement (Madaan et al., 2023) offer trade- off. They do not require pre-computation, but the absence of such computations translates into in- creased generation time. Notably, self-refinement methods (Madaan et al., 2023; Shinn et al., 2023; Gou et al., 2023; Huang et al., 2022) have attracted significant attention due to their simplicity, scala- bility, and adaptability to existing models.
The implications of these advancements for the LLM community are immense. Not only do we provide cost-effective and high-performing alterna- tives, but the feasibility of running LLMs on con- sumer hardware also unlocks plethora of novel applications. For instance, in the gaming industry, developers can now make informed trade-off's be- tween NPC AI response quality and visual fidelity, Moreover, software can locally parse sensitive doc- uments such as emails and generate high-quality responses comparable to proprietary models with- out compromising user privacy.
(Madaan et al., 2023) describes a feedback- driven process wherein the initial output from the model is evaluated across a ten-parameter frame- work, including parameters like Safety, Consis- tency, etc. Despite this approach's 's demonstrable effectiveness, as evidenced by an 8.7% absolute performance improvement for GPT-4 (OpenAT, 2023) over its baseline, it introduces potential bias through external intervention in the reflection process. Consequently, this generates feedback in a specific format that might unduly influence the result benchmark. Similarly, (Shinn et al., 2023) also introduces bias through its utilization of unit
We structure this paper to first, elucidate the challenges, delineate our contributions and subse- quently explore practical applications across dif- ferent domains. Through our research, we shed light on a promising avenue for democratizing the use of powerful language models while reducing costs, preserving privacy and circumventing other constraints of proprietary systems.

---

Supervision-fre Supervision-free
Generalized cridque/tedback
Improvement on various model sizes
Ittrative
refiner
feedback
Prompted Refiners: (Shinn al., 2023; Fu al., 2023)
X
X
X
X
Self-Refine: (Madaan 2023)
Domain- Aanniii-g Reff-eeefeett (this work)
Table Comparison of our work and other related works.
test solutions as model feedback, allowing it to cor- rect its errors within their coding problem suite. (Gou et al., 2023) shows significant improvement in technical tasks with the use of external tools to minimize hallucination and toxicity to provide high quality responses. Though this an effective strat- egy, the use of external tools during the refinement process adds significant overhead costs, thereby posing problem from a consumer and scalability perspective.
leveraged to refine the generated zero-shol response of our model. Domain agnostic self-refinement dif- fers from the prevailing self-refing self-refinement paradigms such (Madaan al., 2023; Shinn et al., 2023) in that abstains from inducing biases in the refine- ment process by precluding the model from access- ing evaluation metrics or incorporating extrinsic data, thereby preventing data leakage or oracle- biased refinement, where the refinement is over-fit to specifically cater to the oracle in question. This allows for the model to explore in unbiased way, allowing it to improve in subtlcties not detectable by a weaker oracle, but noticeable by human observer.
(Huang et al., 2022) employs the LSMI process, which requires generating multiple outputs for task before determining the optimal solution via response analysis. The model then learns from or internalizes its own solution. However, this process demands a higher token limit and additional com- putation requirements. Moreover, (Huang et al., 2022) focuses on 540B parameter model, with attempts replicate the LSMI process on smaller models yielding subpar results, thereby indicating scalability issue. A similar scaling concern is observed in (Madaan al., 2023) in their attempt to replicate results with the Vincuna-13B model (Chiang et al., 2023).
This iterative process primarily consists of three distinct phases: (1) The initial zero-shot response generation of the model for a specified input prompt. (2) The self-critique of the zero-shot re- sponse devoid of external feedback or recently ac- quired out-of-model knowledge. (3) The enhance- ment of the initial response, informed by the results of the second phase.
To eliminate the need for dynamic instruction binding, process where instructions are chosen conditional the input prompt, we introduce three static instructions during model compilation. Izero- for the zero-shot response generation, Teritique to deliver comprehensive, unbounded critique and Teefiner to refine the zero-shot response based on the feedback. In addition, we define Teval to assist in the evaluation of two responses.
Analysis of these drawbacks were the primary motivation for the implementation of domain- agnostic self-refinement. We aim to overcome the above mentioned challenges by proposing do- main agnostic self-refinement. Furthermore, aim to produce hierarchy of different SoTA open source large language models by introducing a scor- ing system (PeRFICS), which, to the best our knowledge, has not been explored in existing liter- ature. This work, therefore, presents a solution that achieves a balance between performance enhance- ment and cost-cfficttvveness, bringing closer to democratizing access to high-perforning language models.
3.1
Domain Agnostic Self-Reiinement
3.1.1 Procedure
Consider tripartite procedure, denoted by P. which comprises an initial zero-shot response gen- eration, a self-critique, and subsequent refinement for a given model M. This procedure mathemat- ically formalized through the subsequent series equations:
3 Method
In this section, we delineate the process of Domain Agnostic Self-Refinement, Specifically, we explore how an generalized critique prompt, and conse- quently, an generalized refinement prompt can be

---

open source models by considering not only their performance but also their computational practi- cality. By encompassing parameters such as the baseline performance, the improvement achieved via refinement, and the cost of inference, PeRFICS offers a multi-dimensioon ment. Furthermore, this ranking metric allows for flexibility and context-relevance by enabling the adjustment of several weights, ensuring its applica- bility across varying computational resources and system requirements. This subsection expounds the mathematical representation and significance of each component within the PeRFICS ranking score, highlighting its potential to bring a paradigm shift in open source model evaluation.
y0 M(ris Inero)
0o M(3gj 90 Teritique)
91 M( y0. co Trefiner)
view on model assess-
where yo denotes the zero-shot response for given task prompt Zij with D representing a task domain, and as the index of the task within the domain containing N tasks. Here, co is the re- sultant self-critique based on the task and zero-shot response, and is the refined response derived from the zero-shot response and self-critique.
3.1.2 Performance Evaluation
To ascertain performance, we generate zero-shot response using control model, denoted as M
3.2.1 Design and Computation of the PeRFICS Ranking Metric
yc = (Tijj Izem)
(1)
The PeRFICS ranking metric is mathematically represented as follows:
We then introduce an oracle, represented by O, which evaluates a specific model response ym against the control response Ve- providing scores 8m and Se, each out of the same predetermined maximum.
V (m) =
exp(r-C(m))+5
(5)
This formula integrates several critical factors into the evaluation of model m, each represented by corresponding symbols. The numerator of the formula emphasises the importance of perfor- mance, while the denominator introduces a penalty factor related to the computational cost of infer- ence.
(Sm, 8c) = ( : yo, Me Leval)
(2)
We further compute the relative score, denoted by 8r,
8m
8,
8c
In the evaluation metric (m)::
The model's refinement for given task domain, denoted as 6, is calculated where STme, represents the relative score for the model and 8 repre- sents the zero-shot generation score for a specific task ().
B(m) represents the model's baseline perfor- mance on the Vicuna benchmark, weighted by a. encapsulating its initial capability.
((m) denotes the model's improvement per- centage post-refinement, weighted by B, in- dicative of its adaptive learning potential.
N 8,
Σ (Sridts 5ramm ==
(4)
N
The total model refinement performance is given by Σx 8; with representing a weighting vector for different task domains, which can be tuned, based on the importance of each domain.
The combined expression 1)- exp(x (a- (BeBBm)) B [(m)) calculates an integrated score, factor- ing in both baseline performance and adaptability. External benchmarks, {m)), employed for cross- validation offer a comprehensive performance as- sessment, weighted by / to capture their signifi- cance.
3.2 PeRFICS: A Versatile, Balanced and Performance-driven Ranking Metric
In this subsection, we present our main contribution the Performance, Refinement, and Inference Cost Score (PeRFICS). PeRFICS, denoted as (m). is a novel, customizable, and comprehensive rank- ing metric, which allows a balanced evaluation of
The denominator exp(r-C(m)) + H incorporates penalty for computational inference cost. Specifi- cally, C(m) is the cost function, with y acting as a discount factor, inversely proportional to computa- tional resource availability. This element balances

---

performance against computational efficiency. 6 stabilizes the metric, shielding low-cost models from excessive penalties.
m
16-Bit (GB)
4-Bit (GB)
Airoboros 7B
14.43
4.44
Vicuna 7B
13.78
4.13
The weights o, 3. p. 1, 7, and ő allow (m) to be customized, ensuring adaptability across vari- ous evaluation frameworks.
Vicuna 13B
27.14
7.41
Alpasta 30B
66.13
12.65
Guanaco 65B
131.50
34.95
4 Experimental Setup
Table 2: VRAM usage for models.
this section, we delineate the design of our exper- imental setup, aiming to evaluate the efficacy of a domain-agnostic self-refinement approach across a diverse suite of state-of-the-art (SoTTA) open-source models of different sizes. The goal of this assess- ment is not only measure the performance im- provement delivered by our method, but also to analyze its cost-effectivenesss, providing compre- hensive insight into the practical implications of our approach. The outcomes will be synthesized into a ranking system, lending a quantitative per- spective to aid comparative analysis of the models.
Mocel Gategory
codre
8-4
20
89
84
-00
common sermo
70
00
terr
7.8
16
71
44
POOC
25
73
10
43
70
20
optoy
3.0
21
23
11
4.1 Model Choice
viuree3
vicree Gloggry
4pesta300
guaraco655
We sourced state-of-the-art (SoTA) open-source models from the Hugging Face OpenLLM leader- board (Beeching, 2023), trusted community plat- form. Based on comprehensive assessment cover- ing reasoning, contextual understanding, and fact- selected models within
Figure 2: % Change in token generation post-refinement.
the Vicuna benchmark facilitates an automatic eval- uation metric. We subject the responses generated by both ChatGPT and our models to our oracle, GPT-4 (OpenAI, 2023) which assigns score rang- ing from 0 to 10 to each response while furnishing elaborate explanation for the assigned score.
checking capabilities, four size categories: 7B, 13B, 30B, and 65B pa- rameters. To ensure a holistic evaluation, we combined quantitative metrics with qualitative in- sights from user experiences across online forums. From this, we identified five models spanning from 7B to 65B parameters for performance eval- uation: Airoboros-7B (jondurbin, 2023), Vieuna- 7B, Ficuma-13B (Chiang al., 2023), GPT4X- Alpasta-30B (MetalX. 2023), and Guanaco-65B (Detmerrs et al., 2023). ChatGPT serves as our performance control, and GPT-4 as our benchmark for reasoning capabilities (OpenAI, 2023).
Our motivation for using GPT-4 as an oracle was influenced by (Chiang and Lee, 2023), where the stability and adaptability of using LLM evalu- ator was demonstrated for open ended story gen- eration and adversarial attack tasks. Research by the Vicuna team (Zheng et al., 2023) shows that GPT-4 acting as a judge on the Vicuna benchmark, achieves over 80% agreement with human evalu- ators which is the same level of agreement be- tween independent human evaluators. (Naismith et al., 2023) also showcase that GPT-4 can evaluate human written discourse in reliable, consistent manner that is in strong agreement with human judgement. This is also echoed in papers such (Hackl al., 2023), where GPT-4 is shown to pro- vide reliable ratings higher education domains, thereby setting a strong lower bound for our evalu- ation procedure.
For in-depth review of the selection criteria, model benchmark scores, and specific evaluation prompts, please refer to Appendices D, A, and C respectively. Appendix F contains a detailed expla- nation regarding model choice.
4.2 Benchmark Choice
this study, the Vicuna Benchmark (Chiang et al., 2023) serves as method employed to assess the qualitative performance of our models. Comprising 80 prompts distributed across distinct categories,

---

Airoboras 7B
Vicuna 7B
Vicuna 13B
GPT4X Alpasta 30B
Guanaco 65B
Task
Zero Shot
Self Relined
Zero Shot
Sell Refined
Zero Shot
Self Relined
Zero Shot
Sell Retined
Zero Shot
Self Refined
Writing
89.91%
98.11%
104.79%
101. 30%
106.80%
94.70%
101. 101.53%
901%
104.98%
Roleplay
94,46%
100.12%
94.24%
102.54%
96.86%
105 25%
98%
103. 103,88%
100.96%
103.15 12%
Commonr-sense
94. 74%
93.65%
102. 16%
116.48%
99 99%
113. 70%
96.32 56%
107. 17%
101. 79%
111 116%
82.53%
67.27%
76.60%
82.29%
92.50%
85.69%
89.25%
105.58%
94.20%
97.25%
Counerfactual
87.92%
96.45%
92.10%
117.49%
99.23%
112.67%
95.23%
112.14%
111.12%
116.68%
Coding
74. 74.35%
59.72%
69 19.42%%
53%%
78. 78.57%
78.08%
84.89%
97.79%
81.6%
90.03%
Math
31.67%
23.33%
31.67%
26.67%
26.67%
33.333%
64.81%
56.85%
53.539
51 51.67%
Generic
92.88%
92.53%
98.01%
112.66%
101 .9%
114.43%
97.09%
100.67%
102.49%
109.65 659%
Knowledge
85.98%
96.91%
95.20%
108.38%
102.29%
110.70%
97.95%
104.11%
100.24%
106.15%
Meun (Ey Weight)
81.60%
79.64%
84.18%
92.96%
88.72%
95.62%
90.28%
98.85%
94.19%
98.99%
Meun (Vicuna)
86.24%
85.31%
89.31%
99.80%
94.53%
101.72%
92.71%
102.57%
98.24%
103.48%
Table 3: Single Refinement Scores a % of ChatGPT Performance.
5 Discussion
lier. Upon careful examination of the responses in this case, it was found that Airoboros' inability to follow instructions contributed to its subpar per- formance. This discrepancy could potentially stem from Airoboros being trained on synthetic dataset, rather than real user interactions. While this might have enabled cost savings during training and im- pressive synthetic benchmark scores (ARC, Hel- laSWAG, MMLU and TruthfulQA), it resulted in severe performance degradation on certain instruc- tion following tasks. A comprchensive analysis of Airoboros' responses can be found in Appendix B.
The experimental results, delineated in Table 3, val- idate our proposed methodology of generalized, dogain-agnostic self-refinement. This table por- trays model scores as relative percentages to Chat- GPT's performance. A notable bias was detected in the oracle's 's evaluation which leaned towards the first response offered for evaluation. To correct this skew, evaluations were performed using unique pairwise orderings, and the final scores were cal- culated as their average. Please consult Appendix (E) for an expanded explanation. Several interest- ing outcomes were observed, which are discussed below:
ChatGPT Comparisons: A comparative study of the open-source models against ChatGPT across varied domains revealed significant differences. a zero-shot setting, Guanaco-65B model displayed the highest average win rate (45.56%) among all models, reflecting its excellent generalization ca- pabilities. Post self-refinement, Guanaco-65B re- tained its supremacy with an impressive win rate of 59.89%. The Vieuna models and GPTtX-Alpasta- 30B also registered considerable improvements, each attaining win rates exceeding 55%.
Effect of Reff-Refinment: Noteworthy perfor- mance enhancements were most prominent in tasks demanding high-creativity, common-sense reason- ing, and counterfactual understanding. As an il- lustration, Vicuna-7B experienced a substantial 14.32% boost in the common sense reasoning task. Similarly, Vicuna- exhibited a significant rise in performance of 13.44% in the counterfactual un- derstanding task. This is likely due to the fact that the opportunity for refinement allows it to evaluate the tokens it generated, re-strategize and provide more context regarding it's stance, thereby facili- tating higher scores from the oracle.
Trends and Domain Limitations: A discernible trend across the self-refinement process was amplification in verbosity. The models tended generate more comprehensive answers, particularly in categories like "Coding" and "Math" 2. How- ever, a higher token count did not necessarily trans- late into improved performance. Despite overall enhancements, not all tasks uniformly benefited from self-refinement. Particularly, performance in math and coding tasks decreased for some mod- els post-refinement. This drop in performance in technical tasks could be attributed several factors including propensity for hallucination, where mod- cls predict statistically likely tokens without truly understanding their semantic relevance, and lack
Performance Enhancement Across Model Sizes: Interestingly, smaller models like Vicuna presented larger percentage improvements compared to their larger counterparts. This could be attributed to their initially lower performance. Nevertheless, it is promising note that models with lesser computa- tional resources can reap substantial benefits from notewor-
the refinement process. However, thy that this trend is not universal among smaller models, with Airoboros 7B being a significant out-

---

% Wins Against ChatGPT Pre-Refiremant
% Wins Against ChatGPT Post-Refinement
Knowlerge
Roleplay
Roleskay
Cosing
Counterfactual
Cooro
Clue bataaat
N033300 ne-15h
Viuren 16 relieed
Apuata relree!
Guarass vofrod
Figure % Wins against ChatGPT for zero-shot responses
Figure 4: % Wins against ChatGPT for refined responses
of self-awareness, where LLMs are incapable of self-correction and improvement due to knowledge deficiency. This hypothesis is strengthend by data that implies that larger models with more param- eters are less likely to degrade on such tasks, im- plying that hallucination and knowledge deficiency reduce with increase model size.
balance of resource usage and performance. Conversely, Airoboros 7B, despite having a higher external average score, ranked last due to its lower baseline and refined performance scores. The Vicuna models demonstrated remarkable performance gains from baseline to refinement, with the 13B variant even surpassing ChatGPT, highlighting the effectiveness of domain agnostic self-refinement approach. Guanaco 65B, while performing admirably in terms of raw power, was ranked lower due to high inference cost. These results underscore the robustness our PeRFICS ranking metric differentiating models by considering their comprehensive performance capabilities, practical computational costs, and the degree of enhancement achieved via refinement. This allows cater broader range of use cases, facilitating informed decision-making for users with varying computational resources and system requirements.
Discoveries: The observations indicate that the efficacy of self-refinement for model depends on multiple factors. These include (1) The capacity for instruction following, where model is capable of both providing effective critique and following instructions refine its output, and (2) The number of parameters in the model, which contributes to stability during refinement, as we notice trends with larger models to resist degradation upon refinement in technical tasks.
Rank
Model
VRAM Cost
Baseline
Refined
Ext-Asg
Alpasta 308
12.65
92.71
102.57
$7.9
Vicuna
3
Vicuna 13B
7.41
94.53
101.72
53.7
4
Guanaco 65B
34.95
98.24
103.48
62.2
The domain agnostic self-refinement approach, when combined with the novel PeRFICS ranking metric, provides a robust framework for augment- ing the performance of open-source LLMs. How- ever, the varied improvement across tasks and mod- els underscores the necessity for model selection based on particular application needs. This method- ology not only offers granular insights into the per- formance various models across different tasks, but also enhances their overall efficacy.
5 Airoburos 4.44
55.60
52.30
79.1
Table 4: PeRFICS ranked models.
According the PeRFICS calcu- lation, Alpasta 30B ranked highest, despite not having the lowest VRAM cost or the best baseline performance. This can be attributed to its signifi- cant improvement through refinement, alongside
PeRFICS:

---

5.1
Case Studies: PeRFICS Application Scenarios
gaming desktop with an NVIDIA RTX 4090 with 24GB VRAM. The optimal model according our ranking metric for this setting is Vieuna 13B, owing to its superior role-play performance and consideration of VRAM constraints.
In this section, we delve into three representative scenarios, each with its unique requirements and constraints, to illustrate the application and advan- tages of PeRFICS. We present specific case studies related email response automation, video game non-player character (NPC) AIs, and corporate lan- guage model (LLM) deployment for code analysis and completion. Despite pushing the limitations of existing hardware, we emphasize the potential benefits as the accessibility to GPUs improves and the LLM community expands, thereby providing broader selection of models to apply PeRFICS to. Through these three case studies, demon- strate the efficacy of PeRFICS in determining the optimal LLM under varying constraints and require- ments. While we have explored a handful of mod- els, we expect the ranking metric to be more benefi- cial as more open source, consumer LLMs become available in the future, providing robust, flexi- ble tool for diverse applications. The selected sce- narios exhibit diverse requirements ranging from privacy concerns and hardware limitations to re- fined performance expectations. We examine each case's constraints and the associated optimal LLM selection based on our ranking metric.
5.1.3 Scenario 3: Corporate LLM Deployment
The final scenario involves corporation intend- ing to deploy an in-house LLM for code analysis and completion, with primary considerations being sensitive data handling and regulatory compliance. This use case operates under the assumption that VRAM cost is not constraint, and the best refined performance in terms of coding is expected. For this high-demand setting, Alpasta-30B emerges as the top performer, showing the highest baseline and refined performance scores for coding tasks.
Conclusion
In conclusion, this study provides pioneering approach enhancing the performance of open- source Large Language Models (LLMs).. Our re- sults, an average performance improvement 8.2% across several models, with an impressive 11.74% and 25.39% boost for the Vicuna-7B model, underscore the efficacy of this approach.
5.1.1 Scenario 1: Email Response Automation In this scenario, we consider software applica- tion designed to automatically generate responses emails on user's device. The primary consid- erations are maintaining privacy, hence ruling out proprietary APIs for secure communications, and ensuring a user- lsserffiendly experience on consumer- grade hardware without slowing down system per- formance. The hardware setting encompasses mid- range laptop equipped with an NVIDIA RTX 3060 graphics card with 12GB VRAM and 16GB sys- tem memory. Our metric, PeRFICS, favours the Vicuna-7B model, which fits the device's hardware constraints and strikes a balance between perfor- mance the writing task and low background load on the device, enhancing user experience.
Alongside this novel refinement methodology, introduced the Performance, Refinement, and Inference Cost Score (PeRFICS), comprehen- sive and adaptable ranking metric that takes into account not just the performance, but also the cost-efficiency and refinement capacity of a model. PeRFICS stands as major contribution to the LLM community, facilitating nuanced under- standing of model performance that allows for optimal model selection based on balanced sessment of performance and resource expenditure. This shift towards a cost-effective, performance- driven ranking system will aid in democratising the use of advanced LLMs, making these powerful tools more accessible across varying contexts and resource availabilities.
5.1.2 Scenario 2: Video Game NPC AIs
This scenario revolves around video game, where the NPC characters are powered by LLMs to el- evate situational awareness and user interaction quality. The primary criteria are exceptional role- play performance and minimal GPU consumption for high graphic fidelity. We consider a powerful
In the broader perspective, our work fosters the growth of the open-source community, presenting method and a tool-set that reduce the reliance on proprictary models, thus paving the way for greater privacy, accessibility, and efsttiffeccivesess.

---

Limitations
els. This includes implementing safeguards against misuse, such as content filters and usage monitor- ing. Additionally, we encourage the continued de- velopment of policies and regulations that promote the ethical use of AT technologies. We also advise that further research be conducted to study the long- term implications of progressive self-refinement in language models.
Our study, while presenting notable advancements in the refinement and selection of language models, is not without its limitations, which we candidly acknowledge and propose to be addressed in future research.
One critical limitation of our study is the re- stricted number of open-souree large language mod- els (LLMs) we were able to test. Our testing was constrained by the high costs associated with using the GPT-4 API, which served as our oracle model, and the high VRAM usage requirements of some larger models. As such, the scope of our study was limited, and the results may not be entirely general- izable to all open-source source LLMs. reiterates the need for cost-effective, open-source alternatives for both model oracles and large-scale LLMss, which would facilitate a more extensive evaluation of self- refinement techniques across diverse models.
In line with the ACL Ethics Policy, we affirm that this study was conducted with the highest stan- dards of ethical research conduct, respecting prin- ciples of honesty, objectivity, integrity, carefulness, openness, respect for intellectual property. confi- dentiality, responsible publication, and responsible mentoring. We also commit to fostering an en- vironment that encourages ethical conduct in Al research and development.
References
In conclusion, while our study contributes mean- ingfully to the fields of LLM refinement and model selection, we recognize the imperative for contin- ued research that extends the reach of our tech- niques, addresses the highlighted limitations, and ultimately advances the field further.
Edward
Beeching. 2023. Open Ilm leader- co/spaces/ HuggingFaacet//ppen. leaderbbard
board. https:
Cheng Han Chiang and Hung-yi Lee. 2023, Can large language models be alternative human evalua- tions? Proceedings of the 61st Annual Meeting of the Association for Compitational Linguistics (Vol- 1: Long Papers), pages 15607-15631, Toronto, Canada. Association for Computational Linguistics.
Ethics Statement
Our research has potential implications from an eth- ical perspective that should be taken into account. Firstly, our work contributes to an equitable distri- bution of Al capabilities by showing that smaller, open-source language models, which are more ac- cessible to the wider community, can compete with larger models in terms of their performance on language generation tasks when domain-agnostic self-refinement is applied. This democratizes ac- cess to high-performing Al technology, bridging the gap between entities with substantial computa- tional resources and those without.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, and Joseph E Gonzalez. 2023. Vicuna: An open-source chatbot impressing chatgpt quality.
gpt-4 with 90%*
A Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury. Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lec, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern Douglas Eck, Jeff Dean. Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways.
However, there are also potential ethical risks associated with our findings. As our techniques enhance the capabilities of open-source language models, these improved models could be utilized for harmful purposes such as disinformation propa- gation or for generating offensive or harmful con- tent. Further, the potentially increased efficiency of these models, in terms of VRAM usage, could inadvertently facilitate misuse by malicious actors, due to the lower computational resources required.
To mitigate these risks, we recommend the re- sponsible use and deployment of the refined mod-

---

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery, Alex Castro-Ros Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang. Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling instruction-finetuned language models.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- nieszka Grabska- Barwinska, Demis Hassabis, Clau- dia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):35213352
Grabska-Barwinska,
Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self feedback.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try are, the ai2 reasoning challenge.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized Ilms.
MetalX. 2023. Gpt--x-alpasta-30ss Merge of al- pacalora with openassistant.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representa- tions.
Ben Naismith, Phoebe Mulcaire, and Jill Burstein. 2023, Automated evaluation of written discourse coherence using GPT-4, In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 394 403, Toronto, Canada. Association for Computational Linguistics.
Jinlan Fu, Kee-Kiong Ng. Zhengbao Jiang, and Pengfei Liu. 2023. Gpiscore: Evaluate as you desire.
OpenAL. 2023. Gpt-4 technical report.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2023. Critic: Large language models can self-correct with tool-interactive critiquing.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal rein- forcement learning.
Veronika Hackl, Alexandra Elena Müller, Michael Gran- itzer, and Maximilian Sailer. 2023. Is gpt-4 a reliable rater? evaluating consistency in gpt-4 text ratings.
Yi Tay, Jason Wei, Hyung Won Chung, Vinh Q- Tran, David R. So, Siamak Shakeri, Xavier Gar- cia, Huaixiu Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, Denny Zhou, Donald Metzler, Slav Petrov, Neil Houlsby, Quoc V. Le, and Mostafa De- hghani. 2022. Transcending scaling laws with 0.1% extra compute.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Mcasuring massive multitask language under- standing.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023, Llama: Open and efficient foundation language models.
Jordan Hoffmann, Sehastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero. Karen Si- monyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training compute-optimal large language models.
Jason Wei, Xuezhi Wang, Dale Schuurmans. Maarten Bosma, Brian Ichter, Fei Xia. Bd Chi, Quoc Lc, and Denny Zhou. 2023. Chain-of-thouuhh prompting elic- its reasoning in large language models.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models.
Rowan Zellers, Ari Holzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence?
Lianmin Zheng, Wei-Lin Chiang, Ving Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric. Xing. Hao Zhang. Joseph E. Gonzalez, and lon Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022. Large language models can self improve.
jondurbin. 2023. Airoboros-7b.

---

A Baseline Performance on External Benchmarks
on the frequency of different types of lightning events (e.g., slou-to-grount vs. intracloud) or highlighting notable instances where particularly large or frequent lightning activity has been observed. Similarly, rather than relying solely on estimates derived from existing datasets, might seek oui additional information from experts in the field or conduct my own research using relevant tools and techniques. By doing so, could potentially gain a deeper understanding of the complex dynamics behind lighming formation and distribution, allowing me to offer a more nuanced and comprehensive response to the original question
Models
Average
HellaSwag
MMLU
TrutfuuQQ
Airoboros-7h
55.6
52.3
79.1
40.1
51.1
Vicuna-7b
52.2.6 6
75.2
37.5
48.9
Vicuna-136
53.7
47.4
78
39.6
49.8
OPT-X-AApasit-300
57.9
56.7
81.4
43.6
49.7
Guanaco 65b
62.2
84.6
52.7
51.3
Table 5: Model Baseline Performance on EleutherAl Benchmarks
B Airoboros
To estimate how many lighining strikes occur on Earth each day, can use data from various sources and make some assumptions based on their findings. According NASA's Light- ning Detection Network (LDN), there are approxi- mately 2,000 thunderstorms occurring worldwide at any given moment. Each storm produces an aver- age of 100 flashes per hour or about 6,000 flashes per day. However, not all these flashes result in di- rect strike to the ground. Some may be indirect hits even miss the target altogether: Therefore, it's safe assume that only around half of them lead successful connection with the surface. This means that roughly 3,000 lightning strikes happen every day. It's important to note that this number could vary depending on factors such as location, time of year, weather patterns, etc. Additionally, measuring exact numbers for something like this would require extensive monitoring equipment and resources which aren't always available. Nonethe- less, I believe my estimation provides reasonable approximation of the daily occurrence of lightning strikes on Earth.
Response 1
The critique provided for Atroboros-7B's initial response ( ('response I'), was positive, confirming the correctness of the intial response. Contrarily, Airoboros-7B seemed to mistakenly perceive this affirming feedback as a directive for supplementary details, including the categorization of different lightning events and incorporation of expert com- mentary or novel research.
This misunderstanding consequently impacted Airoboros- - refinement process of its response. As displayed in 'Refined response', Airoboros-7B tried to include these perceived requests by suggest- ing the addition of more nuanced details about light- ning phenomena-an enhancement not requested in the critique. The outcome not only deviated from the actual intent of the feedback but poten- tially detracted from the clear and concise nature of 'response I', feature praised in the critique. This situation demonstrates the flaw in refinement capability of Airoboros-7B
C
Prompts
The response seems accurate and well- reasoned. It provides clear explanation of the method used to arrive at the estimated figure of 3, 5,000 lightning strikes per day, taking into account potential limitations and uncertainties involved in the calculation. Overall, appears to be a credi- ble response that effectively addresses the question posed.
Critique
IIero: You are tasked with improving the qual- ity of a response to question. The question and responses are provided below. Question:
Ieriique: Reflect on the response. Analyze the correctness of the information provided, the coherence and clarity of the explanation, the depth of the answer given the complexity of the question, and the relevance of your response the specific context of the question. Provide only your critique.
Refined Response- Failed Instruction Following
To improve upon my previous response, would suggest incorporating more specific details and examples related the topic of lightning strikes. For instance, instead of simply stating ëach storm produces an average of 100 flashes per hour. might elaborate further by providing statistics
Trefiner: Based on your initial response and the subsequent self-critique, consider ways in which the response could be improved. Now,

---

provide an enhanced and refined response to the initial question. Give me just the enhanced response.
sampling, which is controlled by Top K set to 40, These settings allow the model only consider a subset of the vocabulary for each token, which is beneficial for creating diverse and grammatically correct responses. Typical P is set to 1, suggesting the model to equally consider all locally typical tokens.
Teval: We would like to request your feedback on the performance of two Al assistants in re- sponse to the user question displayed above. Please rate the helpfulness, relevance, accu- racy, level of details of their responses. Each assistant receives an overall score on a scale of to 10, where higher score indicates better overall performance. Please first out- put single line containing only two values indicating the scores for Assistant I and 2, re- spectively. The two scores are separated by space. In the subsequent line, please provide a comprehensive explanation of your evalua- tion, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment.
Repetition Penalty: We set the Repetition Penalty to 1.18 to discourage repetitive generation. a common issue with language models.
Num Beams: Beam search, controlled by Num Beams, is breadth-first search strategy for gen- eration. We have set Num Beams to 1, indicating no beam search, to allow faster generation the expense the output quality.
Early Stopping: We set Early Stopping to False to allow the model to consider all possible can- didates before stopping, ensuring high-quality re- sponses.
D Hyperparameter Set
D.2 Sampling
this section, we present and explain the rationale behind the specific values selected for various pa- rameters used in our text generation process. The aim is to offer insight into the intentions behind the choices made and to explain the theoretical under- standing that supports these decisions. Below is an elaboration on the key parameters in the context of this work.
Add Bos Token: This means that each input will start with special begimning-of-senteen token. This helps the model understand the context better, leading to improved output quality.
We Add Bos Token to True.
Truncation Length and Skip Special Tokens: We set Truncation Length 2048 and Skip Special Tokens to True. Truncation length limits the size of the imput to prevent overloading the model memory, while skipping special tokens ensures the generated output only includes text tokens, making it easier for downstream tasks.
Max Tokens, Min Length, and Max New Tokens: The generation length has significant implications on the usefulness of the output. We have set the Max Tokens and Max New Tokens both to 1024, which allow the model to generate relatively long passages. This is because we targeting genera- tion of detailed responses or discussions. The Min Length has been set to O, enabling the model to generate short responses when contextually appro- priate.
Chat Prompt Size, Chat Generation Attempts, and Seed: These parameters are related to the chat-like interaction setting of the model. Chat Prompt Size set to 2048, indicating the model keeps this much of the conversation history. Chat Generation Attempts is set to 1, meaning the model will try only once to generate response for each input. Seed is set to -1, ensuring different runs of the model will produce different results.
D.1 Generation Strategy
Temperature:
The Temperature value is set to 0.7. Lower values make the output more deterministic and higher values result in more randomness. We selected 0.7 to strike balance between determin- ism and randomness, thus enabling the model to generate diverse yet coherent responses.
In conclusion, the parameters for this model have been meticulously chosen and tailored to facilitate the generation of detailed, coherent, and diverse text. Future work could investigate the effects of varying these parameters, potentially leading more fine-tuned control of the text generation pro- cess.
Top Top K, Typical P: We use these parame- ters to ensure a mix of randomness and relevance in the generated output. Top P is set to 0.1 to ap- ply nucleus sampling, a dynamic variant of top-k

---

Parameler
Value
showing that these benchmarks should be taken into account but should not overshadow the model's initial performance and improvement capabili- ties.
Max Tokens
1024
Tempurature
0.7
Top
Typical P
Top
Chat Prompt Size
2048
Gieneaation Attenuges
Max New Tokens
1024
D.4 Inference Cost Parameters
Epeilon
Ei Cuioft
The parameters in the denominator of the equa- tion deal with the cost of model inference. The y parameter is set to 0.05, implying that while cost is important consideration, it should not over- whelm the benefits of higher performance. This lower value ensures that we do not unduly penalize more computationally intensive models unless their increased performance does not justify their higher costs.
Panalty
Min Length
Repeat Ngram Size
Nom Beams
Penalty Alpha
Length Penalty
Early Stopping
False
Mirestat Mode
Mirociat Tau.
Mirestat
Sesd
-1
Truncation Length
2048
Eos
Skip Special Tokens
Troe
The 5 parameter was set to very small value (0.00001). The inclusion of this term is primarily to prevent division by zero in scenarios where a model has very low inference cost. Additionally, the presence of á ensures that even very low-cost models are overly advantaged by the cost factor.
Table 6: Hyperparameter Set Generation
The selection of the parameter values for the PeRFICS ranking metric was guided by both empir- ical evidence and theoretical considcrations, seek- ing to balance performance evaluation and com- putational cost in a way that would allow broad range of models to be fairly and effectively com- pared.
D.5 Overall Strategy
The chosen parameters for the PeRFICS metric re- flect an overall strategy of balancing the value performance, refinement capability, and cost. They were selected to ensure that the metric does not overemphasize any one component at the expense of others, thereby maintaining versatile and prac- tical measure of model's 's value across a variety of tasks and environments. The flexibility of these parameters means they can be adjusted as per the specific requirements, making PeRFICS a highly adaptable tool for model evaluation.
D.3 Performance and Refinement Parameters
The parameters a and 3 control the relative impor- tance of the model's baseline performance and im- provement through refinement, respectively. Both set 1, reflecting belief that these two fac- tors should contribute equally to the overall score of model. This balanced approach gives weight both the model's inherent capabilities and its ability to learn and improve over time.
Parameter
Value
The parameter sets the base weight for the ex- ponential term in the numerator, essentially setting the scale for the performance part of the score. We set 77 to 1, implying that the exponential term's out- come will directly contribute to the overall score without any scaling down or up.
0.5
1
0.05
0.5
0.00001
The parameter controls the "steepness" of the exponential function. By setting x to 0.5, we en- sure that both significant increases in performance and more modest improvements are adequately re-
0.5
Table 7: Hyperparameter Set PeRFICS
warded, avoiding an overly aggressive favoring of nigh-performing models that might not offer sig- nificant improvements in practical, real-world context.
E Individual Tables
Below are the model-conditional results (Table 8 15) as measured against ChatGPT and evaluated by GPT-4.
The o parameter determines the weight of exter- nal benchmarks in the ranking. We set p to 0.5,

---

Task
Zero Shot
Self Refined
Change
Task
Zero Shot
Self Refined
Change
Writing
85.55%
82.22%
-3,33%
Writing
94. 24%
91.25%
-3.0%
Roleplay
95.41%
102.22%
+6.81%
Roleplay
93.51%
98.02%
+4.51%
Common Sense
95.90%
93.95%
-1.95%
Common Sense
93.59%
93.35%
-0.24%
Fermi
77.32%
61.82%
-15.5%
Permi
87.73%
72.22%
-15.51%
Counterfactual
88. 19%
97.15%
+8.96%
Counterfactual
87.64%
95.75%
+8.11%
Coding
70.63%
58.95%
-11.68%
Coding
78.06%
60.48%
-17,58%
Math
36.67%
26.66%
-10.01%
Math
26.67%
20.0%
-6677%
Generic
90.56%
90.27%
-0.29%
Generic
95.21%
94.79%
-0.42%
Knowledge
85,44%
98. 18%1
+12.75%
Knowledge
86.51%
95 95.62%
+9.11%
Mean (Eq Weight)
80.63%
79.05% %
-1.58%
Mean (Fq Weight)
82.57 %
80.22%
-2.35%
Mean (Vicuna)
84.85%
84.39 %
-0.46%
Mean (Vicuna)
87.64% %
86.23%
-1.41%
Table Single Refinement Airoboros 7B ChatGPT
Table 12: Single Refinement of Airboros 7B SYS ChatGPT (Scores as % similarity)
Sys (Scores % similarity)
Task
Zero Shot
Self Refined
Change
Task
Zero Shot
Self Refined
Change
Writing
101.11%
106.38%
+5.27%
Writing
101 40%
107.21%
+5.72%
Roleplay
96.94%
104.86%
+7.92%
Roleplay
96. 76%
105 64%
+8.86%
Common Sense
98.47%
107.98%
+9.51%
Common Sense
101.51%
119.42%
+17.91%
Fermi
85.17%
83.60%
-1.57%
Fermi
99.83%
87.78%
-12.05%
Counterfactual
96.87%
109.58%
+12.71%
Counterfactual
101.59%
115.76%
+14.17%
Coding
73.80%
69.55%
-.25%
Coding
83.33%
86.61%
+3.28%
Math
26.66%
33.33%
+6.67%
Math
26.67%
33.33%
+6.66%
Generic
97.63%
106.73%
+9.1%
Generic
104.54%
121.93%
+17.39%
Knowledge
99.72%
108.61
+8.89%
Knowledge
104,85%
112.78%
+7.93%
Mean (Eq Weight)
86.26%
92.29 %
+6.03%
Mean (Eq Weight)
91.18%
98.94%
+7.76%
Mean (Vicuna) 91.95%
98.30% %
+6.35%
Mean (Vicuna)
97.11%
105.14%
+8.03%
Table 9: Single Refinement of Vicuna 13B ChatGPT
Table 13: Single Refinement of Vicuna 13B SYS ChatGPT (Scores as % similarity)
vs Sys (Scores as % similarity)
Task
Zero Shot
Self Refined
Change
Task
Zero Shof
Self Refined
Change
Writing
93.33%
102.5%
+9.17%
Writing
96.07%
100.55%
+4.48%
Roleplay
93.19%
108. 88%
+15.69%
Roleplay
91.36%
98.87%
+7.51%
Common Sense
95.69%
72%
+9.03%
Common Sense
96.95%
109.61%
+12.66%
Fermi
83.33%
99. 10%
+15.77%
Fermi
95.17%
111.99%
+16.82%
Counterfactual
91.80%
110.31%
+18.51%
Counterfactual
98. 68.65%
113.97%
+15.32%
Coding
86.04%
103.45%
+17.41%
Coding
83.73%
92.12%
+8.39%
Math
66.67%
60.37%
-6.3%
Math
62.96%
53.33%
-9.63%
Generic
93.61%
95.41%
+1.8%
Generic
100.57%
105.93%
+5.36%
Knowledge
97.36%
102.77%
+5.41%
Knowledge
98.54%
105.44%
+6,9%
Mean (Eq Weight)
89.00%
98.61 %
+9.61 %
Mean (Eq Weight)
91.56 %
99,09 %
+7.53%
Mean (Vicuna)
91.07%
101.78%
+10.71%
Mean (Vicuna)
94.35 %
103.35% %
+9%
Table 10: Single Refinement of Alpasta 30B ChatGPT vs Sys (Scores as % similarity)
Table 14: Single Refinement of Alpasta 30B SYS ChatGPT (Scores as % similarity)
Task
Zero Shot
Self Refined
Change
Task
Zero Shot
Self Refined
Change
Writing
100.13%
106.73%
+6.6%
Writing
82%
103.23%
-0.59%
Rolcplay
102.08%
103.47%
+1.39%
Roleplay
99.84%
102.77%
+2.93%
Common Sense
98. 98.26%
87%
+8.61%
Commmn Sense
105.31%
116.05%
+10,74%
Fermi
88.88%
91.03%
+2.15%
Fermi
99.52%
103.47%
+3.95%
Counterfactual
105.69%
109.86%
+4.17%
Counterfactual
116.55%
123.49%
+6.94%
Coding
78.96%
87.58%
+8.62%
Coding
84.25%
92.48%
+8.23%
Math
43.33%
43,33%
0%
Math
63.33%
60.0%
-3.33%
Generic
96.45%
103.33%
+6.88%
Generic
108.52%
115.97%
+7.45%
Knowledge
96.80%
103.95%
+7.15%
Knowledge
103.67%
108.35%
+4.68%
Mean (Eg Weight)
90.07%
95.13% %
+5.06 %
Mean (Eq Weight)
98.31 %
102.87% %
+4.56%
Mean (Vicuna) 94.57%
99,94 %
+5.37 %
Mean (Vicuna)
101.89
107.01 %
+5.12%
Table 11: Single Refinement of Guanaco GPT vs Sys (Scores as similarity)
65B Chat-
Table 15: Single Refinement of Guanaco 65B SYS ChatGPT (Scores as % similarity)

---

F Model Selection
We use ChatGPT control to evaluate the per- formance of each model (pre and post-refinement) and GPT-4 as our oracle, due its exceptional rea- soning capabilities (OpenAI, 2023). Please refer to Appendix for the specific prompts used.
In the pursuit of models that exhibit the highest level of baseline performance, our selection process involved comprchensive review of the statc-of- the-art (SoTTA) models listed on the Hugging Face OpenLLM leaderboard (Beeching, 2023), which is an authoritative source in the community as it regu- larly evaluates open source models based on four widely recognized metrics for the evaluation of lan- guage models A12 Reasoning Challenge (ARC) (Clark et al., 2018), HellaSWAG (Zellers et al., 2019), MMLU (Hendrycks et al., 2021), and Truth- fulQA (Lin et al., 2022). These metrics were se- lected for their comprehensive coverage of reason- ing. contextual understanding, and fact-checking capabilities. To ensure the resilience of the do- main agmostic self-refinement process and the sub- sequent PeRFICS metric, we choose open-source models in four popular size brackets 7B, 13B, 30B and 65B parameters. Please refer Appendix D for details regarding each tunable parameter and reasoning behind the values.
To supplement this quantitative analysis, con- ducted qualitative exploration to gauge the real- world applicability and user satisfaction the mod- els. This involved a review of user experiences, opinions, and discussions across various online fo- rums and content aggregators. We also conducted basic testing to get hands-on experience with the models, assessing their functionality from a first- person perspective. The primary aim of this multi- faceted selection process is to scrutinize models beyond their theoretical performance, taking into account practical considerations that influence their use real-world applications. further enables us to identify models that are not just powerful, but also efficient, reliable, and favorably perceived by the user community.
Based the amalgamation of these quantita- tive and qualitative evaluations, we have identi- fied five models that provide high performance across various aspects. The models selected for this study are as follows: Airoboros-7B (jondurbin, 2023), Vicuna-7B, Vicuna-13B (Chiang et al., 2023), GPT4X-Alpasta-30B (MetalX, 2023), and Guanaco-65B (Dettmers et al., 2023). These mod- els vary in size from 7B to 65B, offering a broad spectrum for performance evaluation and compar- ison. Please refer to Appendix for a compre- hensive overview of benchmark scores for these models.


Reviews:
[{"review_id": "PsWjMko8s3", "reviewer": "Reviewer_9ySn", "summary": "", "questions": "1. How to justify the trustworthiness of the testing results if GPT-4 is not perfect? Can the authors give any theoretical analysis for the performance guarantee?\n2. How does the proposed methodology perform in different experiment runs?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 9ySn,\n\nThank you for your helpful comments. We are glad you found our research problem important and ranking metric novel.\n\n\n## **GPT-4 As Our Evaluator**\n\n**TL;DR**: We utilized GPT-4 as an oracle based on its proven reliability in recent literature ([1], [2], [3], [4]). While we acknowledged and addressed GPT-4 biases, such as positional preference ([5]), through methodological adjustments detailed in Appendix E, GPT-4's consistent evaluation behavior minimized the need for multiple runs. Our manual review confirmed GPT-4's accuracy in our context. To strengthen our findings, we propose testing with multiple oracles, including Llama2 and Platypus2. Updates will be incorporated in our revised manuscript.\n\n---\n\n- We understand your concerns about the untrustworthiness of GPT-4 as a strong evaluator. Our motivation for using GPT-4 as an oracle was influenced by [1](https://aclanthology.org/2023.acl-long.870), where the stability and adaptability of using an LLM evaluator was demonstrated for open ended story generation and adversarial attack tasks.\n\n- In extremely recent research by the Vicuna team [2](https://arxiv.org/abs/2306.05685), we see that GPT-4, when acting as a judge for tasks on the Vicuna benchmark, achieves over 80% agreement with other humans - which is the same level of agreement between independent human judges.\n\n- An ACL paper by Duolingo research [3](https://aclanthology.org/2023.bea-1.32.pdf), showcases that GPT-4 can evaluate human written discourse in a reliable, consistent and in a manner that is in strong agreement with human judgment.\n\n- These results are echoed in other papers such as [4](https://arxiv.org/abs/2308.02575), where GPT-4 is shown to provide reliable ratings in domains of higher education, which speaks to the calibre of GPT-4s evaluation prowess.\n\nHowever, this is not to say that GPT-4 is without fault and a perfect evaluator.\n\n- In our experimentation, we did notice a significant positional, ordering and labeling bias, as referenced in Section 5, Line 438. For eg: GPT-4 tended to prefer the first input grading candidate. This issue was later brought up by an independent researcher in Vicuna team's codebase [5](https://github.com/lm-sys/FastChat/issues/826), and was acknowledged by the first author.\n\n- However, this did not impact our research as we already had taken the appropriate steps to mitigate these biases, such as running the experiment multiple times and creating unique pairwise orderings, which were then averaged to mitigate the biases. Appendix E, line 1105 onwards, showcases the base data, pre-aggregation.\n\n---\n\n### **Evaluation Across Different Runs**\n\n- GPT-4's demonstrated consistency across repetitions, as evidenced in [4](https://arxiv.org/abs/2308.02575). The results in this study suggest that with clear prompts and proper system settings, GPT-4 exhibits deterministic behavior. This consistent output not only ensures reliability on the first pass but also potentially eliminates the need of repetitive runs. Due to the high cost of multiple passes through GPT-4, and cost of correcting for the aforementioned bias, we were unable to perform multiple runs.\n\n**Thank you for bringing this to our attention. We will address this as a limitation in our revised manuscript.**\n\n---\n\n### **Manual Review**\nIn order to leave no room for doubt, the authors and a team of researchers have also manually hand-reviewed the responses, the critiques and the refined improvements of each model, to assess the consistency and accuracy of GPT-4, and have reached a general consensus, regarding the overall capability of GPT-4 as an evaluator in this particular use case.\n\n\n**We will revise our paper's discussion section to address the same.**\n\n---\n\n### **Additional Experiment Proposal**\n\nTo further solidify the basis of our work, **we propose the following additional experiment** - where we evaluate the responses with multiple oracles - including, but not limited to Meta’s new Llama2 [6](https://arxiv.org/abs/2307.09288), Platypus2 [7](https://arxiv.org/abs/2308.07317) and other SoTA LLMs that have demonstrated strong performance on the HuggingFace OpenLLM leaderboard [8](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard). Please let us know if this approach with an additional experiment would aid in solidifying the results of our research.\n\n  ---\n\n## **Foundational Concepts and Illustrative Examples**\n\nWe agree with the reviewer that we can improve upon this submission by adding teaser figures and preliminary analysis in our introduction section. We will include an image which will showcase the zero-shot vs refined response examples, and potentially additional case studies as well.\n\n---\n\nWe greatly appreciate your review. Including these changes will create a stronger submission. Please let us know if you've any additional concerns. Thank you once again for your time and in-depth analysis.\n\n## **References**\n\n[1] Chiang, C.-H. et al. (2023) Can Large Language Models Be an Alternative to Human Evaluations?, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Available at: https://aclanthology.org/2023.acl-long.870 (Accessed: 25 August 2023).\n\n[2] Zheng, L. _et al._ (2023) _Judging LLM-as-a-judge with MT-bench and Chatbot Arena_, _arXiv.org_. Available at: https://arxiv.org/abs/2306.05685 (Accessed: 25 August 2023).\n\n[3] Naismith, B. _et al._ (2023) _Automated evaluation of written discourse coherence using GPT-4_. Available at: https://aclanthology.org/2023.bea-1.32.pdf (Accessed: 25 August 2023).\n\n[4] Hackl, V. _et al._ (2023) _Is GPT-4 a reliable rater? evaluating consistency in GPT-4 text ratings_, _arXiv.org_. Available at: https://arxiv.org/abs/2308.02575 (Accessed: 25 August 2023).\n\n[5] Lm-Sys (2023) _Evidence of bias in the ‘fun’ evaluation method using GPT-4 scores? · issue #826 · LM-Sys/FastChat_, _GitHub_. Available at: https://github.com/lm-sys/FastChat/issues/826 (Accessed: 25 August 2023).\n\n[6] Touvron, H. et al. (2023) Llama 2: Open Foundation and fine-tuned chat models, arXiv.org. Available at: https://arxiv.org/abs/2307.09288 (Accessed: 28 August 2023).\n\n[7] Lee, A.N., Hunter, C.J. and Ruiz, N. (2023) Platypus: Quick, cheap, and powerful refinement of llms, arXiv.org. Available at: https://arxiv.org/abs/2308.07317 (Accessed: 28 August 2023).\n\n[8] Open LLM leaderboard - a hugging face space by huggingfaceh4 (2023) Open LLM Leaderboard - a Hugging Face Space by HuggingFaceH4. Available at: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard (Accessed: 28 August 2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The research problem in this work is important. \n2. The proposed method and ranking metric are novel.\n3. The experiments on benchmark datasets including different tasks, and the proposed work shows promising results. \n", "reasons_to_reject": "A few concerns as follows:\n\n1. This work uses GPT-4 as the oracle, but the correctness of the oracle is not guaranteed, which may limit the trustworthiness of the evaluation results.\n\n2. It is suggested to add more preliminaries and examples in introduction to better introduce the background and motivation.\n\n3. To better evaluate the robustness of the proposed methodology, standard deviation or statistical tests for the experimental results are encouraged to be added.\n", "questions_to_authors": "1. How to justify the trustworthiness of the testing results if GPT-4 is not perfect? Can the authors give any theoretical analysis for the performance guarantee?\n2. How does the proposed methodology perform in different experiment runs?", "paper_topic_and_main_contributions": "This work points out some issues such as privacy in Large Language Models (LLMs). It addresses these issues by proposing a generalized variant of self-critique and self-refinement without external influence and a novel ranking metric Performance, Refinement, and Inference Cost Score (PeRFICS) to find the optimal model. The proposed methodology shows some impressive performance improvement in experiments.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nU4ddWsnB2", "reviewer": "Reviewer_7UkX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 7UkX,\n\nThank you for taking the time to read our work and offering your valuable feedback.\n\n\n## **Paper Focus and lack of results from self refine**\n\n  \n\n- You are accurate in recognizing that our method is a variant of Self Refine [9](https://arxiv.org/abs/2303.17651) (as mentioned in our manuscript - lines 209 - 212). We would like to use this opportunity to elaborate upon our core motivations for this work, which were not only to design this domain agnostic framework, but also to devise a metric (PeRFICS) to rank the “refinability” of SoTA open source models of varying sizes.\n\n### **How our method differs from self refine**\n\n- Self Refine follows a targeted approach for refinement and critique (Madaan et al, Page 31, Appendix M.1) [9](https://arxiv.org/abs/2303.17651), where up to six in-context examples are passed in to assist with feedback process. This can also be seen in their attached code base on Github [10](https://github.com/madaan/self-refine/blob/main/src/commongen/data.py), where very fine-grained feedback examples are provided to the feedback generator, thereby inducing a bias in the refinement process.\n\n- The ability of models to improve from in-context prompting is an emergent property of larger LLMs (GPT-3 and bigger) [11](https://arxiv.org/abs/2303.03846), as demonstrated by a Google Brain research paper. **This finding is replicated by the Self Refine paper [9](https://arxiv.org/abs/2303.17651), where they explicitly state that they were not able to replicate their experimental results for smaller models, specifically Vicuna 13B (Madaan et. al, Page 7, Section 4 - Does Self Refine Work With Weaker Models)** [9](https://arxiv.org/abs/2303.17651).\n\n- They acknowledge that their experiments failed to produce consistent results for Vicuna 13B. Due to this acknowledgement and high costs associated with repeated experimentation, we decided to not extensively test our method against Self Refine. **We acknowledge this as a limitation of our study, and we will revise our limitations section to reflect this. Thank you for your constructive feedback.**\n\n- However, due to the domain agnostic nature of our refinement process, which does not harness the use of in-context prompts to guide the feedback engine driving the ratings, we are able to achieve SoTA refinement results **without in-context prompting**.\n\n_To reiterate, our core motivation was to demonstrate the consumer applicability of refined open source LLMs (as evidenced by our case studies) which are inherently of smaller sizes, thereby allowing researchers and users to make educated decisions with regard to model selection using PeRFICS._\n\n\n## **GPT-4 As Our Evaluator**\n\n**TL;DR**: We utilized GPT-4 as an oracle based on its proven reliability in recent literature ([1](https://aclanthology.org/2023.acl-long.870), [2](https://arxiv.org/abs/2306.05685), [3](https://aclanthology.org/2023.bea-1.32.pdf), [4](https://arxiv.org/abs/2308.02575)). While we acknowledged and addressed GPT-4 biases, such as positional preference ([5](https://github.com/lm-sys/FastChat/issues/826)), through methodological adjustments detailed in Appendix E, GPT-4's consistent evaluation behavior minimized the need for multiple runs. Our manual review confirmed GPT-4's accuracy in our context. To strengthen our findings, we propose testing with multiple oracles, including Llama2 and Platypus2. Updates will be incorporated in our revised manuscript.\n\n---\n\n- We understand your concerns about the untrustworthiness of GPT-4 as a strong evaluator. Our motivation for using GPT-4 as an oracle was influenced by [1](https://aclanthology.org/2023.acl-long.870), where the stability and adaptability of using an LLM evaluator was demonstrated for open ended story generation and adversarial attack tasks.\n\n  \n\n- In extremely recent research by the Vicuna team [2](https://arxiv.org/abs/2306.05685), we see that GPT-4, when acting as a judge for tasks on the Vicuna benchmark, achieves over 80% agreement with other humans - which is the same level of agreement between independent human judges.\n\n  \n\n- An ACL paper by Duolingo research [3](https://aclanthology.org/2023.bea-1.32.pdf), showcases that GPT-4 can evaluate human written discourse in a reliable, consistent and in a manner that is in strong agreement with human judgment.\n\n  \n\n- These results are echoed in other papers such as [4](https://arxiv.org/abs/2308.02575), where GPT-4 is shown to provide reliable ratings in domains of higher education, which speaks to the calibre of GPT-4s evaluation prowess.\n\n  \n\nHowever, this is not to say that GPT-4 is without fault and a perfect evaluator.\n\n  \n\n- In our experimentation, we did notice a significant positional, ordering and labeling bias, as referenced in Section 5, Line 438. For eg: GPT-4 tended to prefer the first input grading candidate. This issue was later brought up by an independent researcher in Vicuna team's codebase [5](https://github.com/lm-sys/FastChat/issues/826), and was acknowledged by the first author.\n\n  \n\n- However, this did not impact our research as we already had taken the appropriate steps to mitigate these biases, such as running the experiment multiple times and creating unique pairwise orderings, which were then averaged to mitigate the biases. Appendix E, line 1105 onwards, showcases the base data, pre-aggregation.\n\n  \n\n---\n\n  \n\n### **Manual Review**\n\nIn order to leave no room for doubt, the authors and a team of researchers have also manually hand-reviewed the responses, the critiques and the refined improvements of each model, to assess the consistency and accuracy of GPT-4, and have reached a general consensus, regarding the overall capability of GPT-4 as an evaluator in this particular use case.\n\n  \n  \n\n**We will revise our paper's discussion section to address the same.**\n\n---\n\n\n### **Additional Experiment Proposal**\n\n  \n\nTo further solidify the basis of our work, **we propose the following additional experiment** - where we evaluate the responses with multiple oracles - including, but not limited to Meta’s new Llama2 [6](https://arxiv.org/abs/2307.09288), Platypus2 [7](https://arxiv.org/abs/2308.07317) and other SoTA LLMs that have demonstrated strong performance on the HuggingFace OpenLLM leaderboard [8](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard). Please let us know if this approach with an additional experiment would aid in solidifying the results of our research.\n\n---\n\n## **Figure Signposts and Formatting**\n \nThank you for pointing out the lack of in-text references and explanations for the figures.\n\n- Figure 1 has negative numbers, as in our experimentation, smaller models have a tendency to be more terse in their responses during the second iteration - post critique. We have explained this phenomenon (Lines 493 - 497), but, as you rightfully have pointed out, we have not explicitly explained the negative values in the image, nor added a linked reference to the same.\n\n- While we have drawn upon inferential statistics in our discussion from Figures 2 and 3, we seem to have not referenced them explicitly.\n\nThank you for your keen observation and notice of this oversight. We will be sure to add explanations for the negative values in Figure 1, and relevant linked references for the figures 2 and 3, where we explain observed phenomena.\n\n  \n---\n\n## **References**\n\n  \n\n[1] Chiang, C.-H. et al. (2023) Can Large Language Models Be an Alternative to Human Evaluations?, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Available at: https://aclanthology.org/2023.acl-long.870 (Accessed: 25 August 2023).\n\n  \n\n[2] Zheng, L. _et al._ (2023) _Judging LLM-as-a-judge with MT-bench and Chatbot Arena_, _arXiv.org_. Available at: https://arxiv.org/abs/2306.05685 (Accessed: 25 August 2023).\n\n  \n\n[3] Naismith, B. _et al._ (2023) _Automated evaluation of written discourse coherence using GPT-4_. Available at: https://aclanthology.org/2023.bea-1.32.pdf (Accessed: 25 August 2023).\n\n  \n\n[4] Hackl, V. _et al._ (2023) _Is GPT-4 a reliable rater? evaluating consistency in GPT-4 text ratings_, _arXiv.org_. Available at: https://arxiv.org/abs/2308.02575 (Accessed: 25 August 2023).\n\n  \n\n[5] Lm-Sys (2023) _Evidence of bias in the ‘fun’ evaluation method using GPT-4 scores? · issue #826 · LM-Sys/FastChat_, _GitHub_. Available at: https://github.com/lm-sys/FastChat/issues/826 (Accessed: 25 August 2023).\n\n  \n\n[6] Touvron, H. et al. (2023) Llama 2: Open Foundation and fine-tuned chat models, arXiv.org. Available at: https://arxiv.org/abs/2307.09288 (Accessed: 28 August 2023).\n\n  \n\n[7] Lee, A.N., Hunter, C.J. and Ruiz, N. (2023) Platypus: Quick, cheap, and powerful refinement of llms, arXiv.org. Available at: https://arxiv.org/abs/2308.07317 (Accessed: 28 August 2023).\n\n  \n\n[8] Open LLM leaderboard - a hugging face space by huggingfaceh4 (2023) Open LLM Leaderboard - a Hugging Face Space by HuggingFaceH4. Available at: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard (Accessed: 28 August 2023).\n\n\n[9] Madaan, A. _et al._ (2023) _Self-refine: Iterative refinement with self-feedback_, _arXiv.org_. Available at: https://arxiv.org/abs/2303.17651 (Accessed: 25 August 2023).\n\n[10] Madaan, A. (2023) _Self Refine Code Repository (Github)_, _GitHub_. Available at: https://github.com/madaan/self-refine/blob/main/src/commongen/data.py (Accessed: 25 August 2023).\n\n[11] J. Wei et al., “Larger language models do in-context learning differently,” arXiv.org, https://arxiv.org/abs/2303.03846 (accessed Aug. 28, 2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper addresses a valuable question regarding the self-refinement of open-source large language models (LLMs).", "reasons_to_reject": "- It appears that this paper merely employs a simplified version of the Self-Refine method (https://arxiv.org/abs/2303.17651) initially devised for GPT, and applies it to open-source LLMs, without presenting any substantial differentiation.\n- Recent research indicates a considerable discrepancy between the auto-evaluation methods based on GPT-4 and human evaluations, thus casting doubt on the reliability of the evaluation method used in this paper and the persuasiveness of its experimental results.\n- In Section 5, the authors present Figure 1, Figure 2, and Figure 3, yet fail to provide explanations or analysis for them. The authors need to clarify why there are negative numbers in Figure 1, as well as the implications of Figure 2 and Figure 3.\n- This paper's methodology has a strong link to Self-Refine, but it lacks results from Self-Refine on the Vicuna benchmark in Section 5.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "- This paper delves into an investigation of cost-performance compromises in autonomous improvement of open-source large language models (LLMs).\n- The authors propose a domain-agnostic self-enhancement method that evades external interferences, along with a new evaluation measure named PeRFICS, which strives to identify the best model for a given task, keeping in mind both performance and cost.\n- Empirical findings from the Vicuna benchmark reveal that this domain-neutral self-enhancement method can assist open-source LLMs in enhancing the quality of their outputs and exceed ChatGPT in performance across most task domains. Additionally, this paper provides three case studies illustrating how effectively PeRFICS can guide users in selecting the most suitable model.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MXLEgBK4ly", "reviewer": "Reviewer_27dr", "summary": "", "questions": "In line 931, you mentioned \"Question\". However, I don't see any of that. Do I miss anything?", "limitations": "", "rating": 3, "confidence": 1, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 27dr,\n\nThank you for your thoughtful review and recognition of the innovative aspects of our work. We appreciate the opportunity to address your concerns.\n\n\n---\n\n\n### **Implementation Details in Section 3.1.1**\n\n  \n\n- We understand that your primary concern lies in implementation details outlined in Section 3.1.1. The goal of the aforementioned section is to provide an algorithmic background into our domain-agnostic refinement approach, and therefore, in order to not detract from the primary focus, we decided to include the relevant fine-grained implementation details in Appendix C and Appendix D.\n\n- Here, we detail the exact prompts used and the model responses. We also specify the hyperparameters used for token generation and the tunable parameters for our model ranking according to the PeRFICS metric.\n\n  \n\nHowever, we recognize your concerns, and will ensure that we add additional signposts referencing the appendix (similar to the one on line 386).\n\n  \n\n**Do you think that adding more signposts and references to the relevant implementation details would address your concerns effectively?**\n\n  \n\n---\n\n  \n\n### **Clarification on Line 931**\n\n  \n\n- As detailed in Section 3.1.1, after instruction _I_zero_, we pass in the necessary row of our benchmark dataset, which is a question.\n\n- Therefore, in order to prime the model (so that it knows what to expect and can answer the question effectively), we have the string “Question:” as our token leading to the appended question. We hope this clarifies your concern regarding line 931's open “Question:”.\n\n  \n  \n\n**We'll ensure clearer exposition of this in our revised manuscript. We appreciate your attention to detail and hope this addresses your concern.**\n\n  \n\n---\n\n  \n\n### **Reproducibility Concerns**\n\n  \n\n- The questions used, the model responses, the oracle’s evaluation and the code necessary to generate the same are provided in the additional supplementary materials.\n\n- We do not use any external, non-public datasets in effort of maintaining the highest standards of reproducibility.\n- We will also move some of the implementation details from the appendix back to the main section to aid this effort.\n\n  \n\n**Could you provide us with some specific areas within our codebase or implementation details where you have reproducibility concerns? We would be glad to clarify or improve accordingly.**\n\n  \n\n---\n\n  \n\n_Please do let us know if you have any further questions or concerns regarding the soundness or reproducibility of our work, and we shall be more than happy to address them._\n\n  \n\n---\n\n  \n\nThank you once again for your insightful comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Innovative approach: The domain-agnostic self-refinement approach proposed in this paper provides a new method to enhance the performance of open-source LLMs without relying on external feedback or pre-training.\n2. Comprehensive ranking metric: The PeRFICS metric takes into account multiple factors, including baseline performance, refinement improvement, and inference cost, providing a balanced evaluation of different models.\n3. Experimental validation: The experimental results demonstrate significant performance improvements achieved through self-refinement across various models of different sizes. The case studies further illustrate the practical applicability and benefits of PeRFICS in different scenarios.", "reasons_to_reject": "My main concern is the limited implementation details: The paper does not provide sufficient implementation details for reproducing the study. It might be useful if more details are provided in section 3.1.1. I find it a little bit hard to figure out the exact procedure of domain-agnostic self-refinement after reading 3.1.1.", "questions_to_authors": "In line 931, you mentioned \"Question\". However, I don't see any of that. Do I miss anything?", "paper_topic_and_main_contributions": "It is an important task to make open source LLM on par with proprietary models. This paper proposes a domain-agnostic self-refinement approach to enhance the performance of open-source Large Language Models (LLMs). The authors also introduce a novel ranking metric called Performance, Refinement, and Inference Cost Score (PeRFICS) to evaluate and compare the performance and cost-effectiveness of different models. The experimental results demonstrate significant performance improvements achieved through self-refinement, with an average improvement of 8.2% across various models. The PeRFICS metric provides a comprehensive evaluation of models, taking into account their baseline performance, refinement improvement, and inference cost. The paper also presents three case studies in email response automation, video game non-player character (NPC) AIs, and corporate LLM deployment, showcasing the applicability and benefits of PeRFICS in different scenarios.\n\nOverall, the paper presents an innovative approach and a comprehensive ranking metric for enhancing the performance of open-source LLMs. The experimental results demonstrate the effectiveness of the proposed self-refinement approach, and the case studies illustrate its practical applicability. However, there are some limitations and areas for improvement, such as providing more implementation details.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a domain-agnostic self-refinement approach with a novel ranking metric to enhance the performance of open-source Large Language Models (LLMs) that evades external interferences. The paper also performs extensive evaluations 

Pros:
- The paper is solving an important and relevant problem
- The ranking metrics are novels and comprehensive
- Promising results on benchmark datasets across multiple tasks.
- The authors have provided code

Cons:
- Unclear description of experiment results 
- Bias in using GPT-4 as a evaluator
- Empirical comparison to self-refine. The authors have compared to self-refine in introduction and related work but comparison to self-refine in experiments is missing
Decision: Accept-Findings