Prompt Engineering Techniques for Language Model Reasoning Lack Replicability
Authors: Laurène Vaugrante
Mathias Niepert
Thilo Hagendorff
Venue: Accepted by TMLR ()

Abstract: As large language models (LLMs) are integrated into everyday applications, research into prompt engineering techniques (PET) to improve these models’ behavior has surged. How- ever, clear methodological guidelines for evaluating these techniques are lacking. This raises concerns about the replicability and generalizability of the prompt engineering techniques’ benefits. We support our concerns with a series of replication experiments focused on zero- shot prompt engineering techniques purported to influence reasoning abilities in LLMs. We tested GPT-3.5, GPT-4o, Gemini 1.5 Pro, Claude 3 Opus, Llama 3, Vicuna, and BLOOM on the chain-of-thought, EmotionPrompting, Sandbagging, Re-Reading, Rephrase- and-Respond (RaR), and ExpertPrompting prompt engineering techniques. We applied them on manually double-checked subsets of reasoning benchmarks including Common- senseQA, CRT, NumGLUE, ScienceQA, and StrategyQA. Our findings reveal a general lack of statistically significant differences across nearly all techniques tested, highlighting, among others, several methodological weaknesses in previous research. To counter these issues, we propose recommendations for establishing sound benchmarks, and designing rigorous exper- imental frameworks to ensure accurate and reliable assessments of model outputs.

Published in Transactions on Machine Learning Research (12/2025)
Prompt Engineering Techniques for Language Model Reasoning Lack Replicability
Laurène Vaugrante
laurene. wwi.raneeeurrrnnneeeei.i...s www.:errrritnittttttrrr..r..
Interchange Forum for Reflecting on Intelligent Systems University of Stuligart
Mathias Niepert
Institute for Artificial Intellignnce University of Stuttgort
Thilo Hagendorff
Interchange Forum for Refeccing on Intelligent Systems
University of Studigort
Reviewed on OpenReview: https: optp://veeerreeee..ommmo net/ forum? id- b92R5bM44u
Abstract
As large language models (LLMs) are integrated into everyday applications, research into prompt enginecring techniques (PET) to improve these models' behavior has surged. How- ever, clear methodological guidelines for evaluating these techniques are lacking. This raises concerns about the replicability and generalizability of the prompt engineering techniques' benefits. We support our concerns with a series of replication experiments focused on zero- shot prompt engincering techniques purported to inffuence reasoning abilities in LLMs. We tested GPT-3.5, GPT-4o, Gemini 1.5 Pro, Claude 3 Opus, Llama 3, Vicuna, and BLOOM on the chain-of- thain-of-thouuht, Sandbagging, EmotionPromppcnng Re-Reading, Rephrase- and-Respond (RaR), and Exppertrompttin prompt engineering techniques. We applied them on manually double-checked subsets of reasoning benchmarks including Common- senseQA, CRT, NunGI.UF, ScienceQA, and StrategyQA, Our findings reveal a general lack of statistically significant differences across nearly all techniques tested, highlighting, among others, several methodological weaknesses in previous research. To counter these issues, propose recommendations for establishing sound benchmarks, and designing rigorous exper- imental frameworks to ensure accurate and reliable assessments of model outputs.
1 Introduction
The field of generative artificial intelligence has considerably evolved in only a few years, In particular, large language models (LLMs) have witnessed an unprecedented surge in popularity with the release of ChatGPT [OpenAl 2022). which became the most rapidly adopted internet application in history. LLMs possess advanced natural language processing capabilities which demonstrate broad range of downstream applications, ranging from casual conversations to complex problem-solving (Minaee al. 2024 Zhou al. 2020). Given the fast growing range of applications Guo al.) 2024 plus their respective risks for Al alignment (Ji et al.] 2024), fairness (Hao et al. 2023 and safety (Amodei et al. 2016 Hagendorff 2024
Vaugrante et al.) 2025 Weidinger et al. 2023) it is paramount to evaluate behavioral and reasoning patterns these models exhibit (iix & Schulz, 2023 Gao et al. 2025 Wang el. al. 2024a). This created the need for new research fields, and fostered a variety of different approaches to investigate different types of LLM
behavior, including emergent abilitics and prompt engincering strategics (Chang al. 2023). substantial part of this research relies on prompt engineering techniques, which are designed steer LLMs toward desired responses without modifying their internal structure (Liu et al. 2021 However, prior work has

---

Published in Transactions on Machine Learning Research (12/2025)
demonstrated that some of these techniques fail to replicate in newer, reasoning-oriented models, particularly in software-engineering contexts [Wang et al. 2024bp. We hypothesize that such replication challenges may extend beyond reasoning models and occur even with instruction-followingg LLMs across a broader range of reasoning tasks, indicating a deeper replication problem within prompt-engineering research. To investigate this, we conduct experiments attempling to conceptually replicate studies investigating zero- shot prompt engincering techniques that are believed to enhance reasoning in LLMs. Our findings reveal that these techniques often fail produce consistent improvements. Based on our findings, we propose specific recommendations and tools for developing better methodologies when evaluating LLM behavior. This includes establishing sound benchmarks, designing robust experimental frameworks adapted to the LLMs used, and implementing accurate evaluations of model outputs.
2 Methods
2.1
Prompt Engineering Techniques
For our experiments, we tried to replicate single-pass, zero-shot prompt engineering techniques that were demonstrated to alter reasoning performances in LLMs in previous studios:
Zero-shot chain-of-thoughtt Prompting (Kojima et al. 2022 : This method claims that adopting
a step-by-step reasoning approach in LLMs enhances overall reasoning accuracy.
Sandbagging (Pere al. back dialog user's preferred response and mirror them when solving tasks.
2022 Sandbagging showcases that LLMs have tendency to repeat
EmotionPrompting (Li et al. 2023 This technique consists in adding emotional stimuli, such as "This very important to my career in order to enhance the performance.
Re-Reading [u ct al. 2024 This method consists in repeating the task twice to enhance the reasoning accuracy.
Rephrase-and-Respone (Deng et al., 2024): This approach involves rephrasing the given task as a query before providing a response, thereby improving accuracy.
ExpertPrompting (Xu et al. 2025): This technique claims to enhance the LLM accuracy when setting the LLM in an expert role.
With this selection, we aim at covering the majority of prompt engineering techniques that represent zero- shot, single-pass methods which are most likely to be adopted by a wide range of users in real-world LLM settings. Furthermore, these techniques simplify the experimental design and minimize potential sources of error, which would be present in more complex settings. We deliberately excluded methods such ensembling, self-criticism,, least-to-most. prompting, tree-of-thoughts,, elc., these involve more complex setups prompt chains (Schulhoff et al. 2024 Yao et al., 2023) which consider unlikely to be utilized by typical LLM users. As prompt engineering techniques can be applied to various applications (Liu et al. 2021h like improving text quality, enhancing the answer's relevance or controlling the output formatting, we chose to focus on prompt engineering techniques used to enhance reasoning abilities and hence accuracy of LLMs when prompted with complex tasks.
2.2 Benchmark Selection
To replicate the claimed impact of the selected prompt engineering techniques on LLM reasoning abilitics, we selected five different benchmarks, each measuring a different. lype of reasoning: Commonsensee TTammor
et al. 2019), CRT (Hagendorff et al., 2023), NumGLUE (Mishra et al. 2022), ScienceQA (Lu et al. 2022 and
Strategy QA GGeaa al.) 22021. In accordance with a growing body of rescarch (Gema et al.] 2024 Goetze
& Abramson 2021) we noticed low quality of many benchmark items, meaning incorrect or ambiguous questions, formatting flaws, factual errors in the response choices. Therefore, we chose to hand-pick
2

---

Published in Transactions on Machine Learning Research (12/2025)
(through rule-based filtering and manual checks) 150 faultless questions out of random sample of 200 questions per benchmark, with total of = 750, preferring accuracy over large sample sizes. The tasks were either open-ended, Boolean, or multiple-choice questions.
2.3 Experiments
We first measured the accuracy of LLMs in a base lest using unmodified lasks. We then applied the prompt engineering techniques outlined the studies mentioned above by incorporating the necessary pre- or suffixes to each task. We used the same prompts described in these studies when available, and generated new ones based the prompt descriptions when they were not. When the studies used several pre- suffixes basis to their claim, such as in the EmotionPrompting study where 11 different emotional stimuli were used, we randomly selected one of them for cach task using a seed.
2.4 LLM selection
We compared the performance of five different LLMs, particular OpenAl's GPT-3.55 (gpt-3.5-turbo
(OpenAT 2022) and GPT-40 (gpt-4o-2024-05-133 (OpenAT 2023), Google's Gemini 1.5 Pro (gemini-1.5-ph)
(Anthropic) 2024). Google's (Team et al. 2024), and Meta's Llama 3, with both 8B and 70B versions (34). As the selected studies also used some models from carlier generations, we have also attempted to mcasure the performance for Vicuna 13B v1.5 CChiang et al. 2023 as well as BLOOM 176B (Workshop et al. 2023). but the obtained results were deemed unusable due to the models' inability to generate coherent outputs, their tendency to produce meaningless loops, repeated fragments the input, and other issues, as detailed in Appendix A
001) (Team al. 2024 Authropic's Claude 3 Opus [claude-3-opu--2024402
Gemini 1.5 Pro (gemini-1.5-pro--
(Meta-Llama-3 Instructt and Meta-Llama-3 70B Instrust
2.5 Output classification
To facilitate the LLM output classification process without restricting the reasoning behavior during the LLMs' prompt completions, added instruction to write the final answer after a specific string, namely " #####" to each benchmark lask, as indicated in the literature (Cobbe et. al.] 2021 Nezhurina et al. 2024). We then assessed the LLM outputs following # # # by combining string matching methods, LLM-based evaluations with GPT-4o, as well as manual double-checks (sce Appendix B). Considoring that the behavior of LLMs might exhibit variations over time (Chen et al. 2024a), we report the timeframe of the experiments. They spanned from June 6th, 2024, to June 17th, 2024, except for the Vieuna 13B and BLOOM experiments, which spanned from December 4th to December 14th, 2024. For all experiments, LLM temperature parameters were set to 0, or 0.00001 when 0 was not permitted.
2.6 Study focus
replication involves collecting and analyzing new data to replicate the findings of a previously conducted study, whereas reproducibility entails reanalyzing the original data to verify the results. Our hypothesis when replicating the previous experiments was that the claimed performance improvements are not replicable and hence the claims about the prompt engineering techniques are not generalizable. We neither use the exact same selections of benchmarks nor models in the original studies but vary the experimental setups slightly. In detail, this means that still test foundation text-to-text models, use reasoning benchmarks, and use prompt engincering techniques cither verbatim (chain-of-thought, Ro-Reading, EmotionPromptingg Rephrasc-and- Respond), or, if necessary, adapted in alignment with their original methodology to suit our benchmarks (Sandbagging, (xspertPrompting.) which should theoretically increase their utility. However, it is important note that our approach to classifying LLM responses likely differs substantially from the original studics, as many of them lack descriptions of their chosen methods, further underscoring the nature of our work replication study rather than reproduction.
This study focuses on replication rather than on reproducibility. According to Peng 2011
3

---

Published in Transactions on Machine Learning Research (12/2025)
2.7 Statistics
All statistical analyses were performed using Python (version 3.11.4). The SciPy library (version 1.13.1) was used for statistical computations, while visualizations were created with Matplotlib (version 3.7.1) and Scaborn (version 0.12.2). We applied chi-squared (x²) tests to assess the statistical significance of accuracy differences belween baseline and modified prompts. Rounded P-values are reported for each test. 95% confidence intervals (CIs) were calculated and included in the result visualizations. Performance variability across models and benchmarks was accounted for, and results were reported per model, benchmark, and prompt engineering technique.
3 Results
3.1 Chain-of-thought prompting
Chain-of-though prompting involves decomposing a given lask and solving each step before outputting the final answer, by presenting the LLM with example of task and its expected decomposed output. In the original study establishing this method, Wei et al. (2023) tested five LLMs over three reasoning, categories including arithmetic reasoning, commonsennse reasoning, and symbolic reasoning, harnessing 12 different benchmarks. The authors claim a good robustness of this method, with several different annotators. While they reported variance in the average performance, was consistently superior to the performance with the basc ovaluation, with a reported average improvement of 39.91% (Wei et al., 2023). A subsequent study then claimed that a zero-shot chain-of-lhought prompting strategy sufficed to elicit similar improvements (Kojima et al. 2022). Instead of presenting, before each task, an example enabling chain-of- oftthughhttht reasoning, they simply suffix tasks with "Let's think step by step". They tested larger sample of 17 LLMs on various reasoning categories, utilizing 12 benchmarks akin to the previous paper. They obtained an averaged 35.93% improvement in accuracy for zero-shot chain-of-thought reasoning across all benchmarks and models (Kojima itt al. 2022). We tried to replicate these findings with our set of reasoning benchmarks. However, despite the impressive results from the original studies, we observed that there was no significant improvement (see Figure with the exact same task suffix as in the original study, we could not observe any significant difference across all benchmarks. With results from all models combined, the maximal positive impact of chain-of-thought reasoning is with NumGLUE where there is 2.78% accuracy difference between the base and the chain-of ohthught prompt (see Appendix q. which is not significant given the total number of tasks (x² = 1.78, p- = 0.18). These numbers remain similar throughout each LLM evaluated, with an overall average improvement of 0% for the chain-of-tie tfotughhtt reasoning (x2 = 0.06, = 0.8), as seen in Appendix C The largest observed positive impact of chain-of-thought reasoning is for Llama 3-70B tasked by Commonsense(AA, with an observed 8.67% improvement (x2 2.19. p- 0.14) (see Appendix(C.. However, the highest overall difference is an 11.33% accuracy decrease (x² 4.47, p < .05) (see Appendix C with chain-of-thought reasoning applied on Llama 3-70B with StralegyQA. While the latest models seem to implement chain-of-thouuht reasoning by default, meaning without being specifically prompted to, these results hold even for previous models such as GPT-3.5, which often do not. We compared the average response length of each LLM when chain-of-thouue th-uught reasoning is explicitly requested, compared to when it is not, as shown in Appendix D Even when the base experiments do not demonstrate verbose prompt. completions and the Mhain-of-thoughi prompting does, the performance results are not impacted in a significant manner, which stands contrary to what the lilerature suggests (Jin et al. 2024). For instance, GPT-4o had an average difference of response lengths of 531 characters for the base test vs. 931 characters for the shain-of-thougle prompting, but just 0.01% accuracy difference, suggesting that simply increasing the length of prompt completions does not enhance accuracy beyond a certain point.
3.2 Sandbagging
Perez et al. 2022) demonstrate sycophancy, which an LLM's tendency to output answers that users tend to prefer. The rescarchers evaluated several aspects of sycophancy, including "sandbagging" capability, which suggests that a model could underperform when a user is deemed incapable to solve or verify a given task. They underpin this hepot by adding user biographies before reasoning tasks from TruthfulQA (Lin et al.
4

---

Published in Transactions on Machine Learning Research (12/2025)
2022,, with "very educated" users as opposed to "very uneducated" users. They imply a significant difference between these two cat categories, claiming that sandbagging causes L.Mss to output incorrect answers when human users are perceived as unable to answer correctly themselves (Perez et al. 2022). We conceptually replicate this experiment using our selected models by prefixing our selected reasoning tasks with both "very educated" and "very uneducated" user biographies (see Appendix E We observe no significant difference over all benchmarks when comparing the highly educated (x2 = 1.64, 0.20) or poorly educated (x² = 1.24, = 0.27) user prompts to the base results (see Figure | and Appendix C with an average accuracy decrease of 1% for both cases (see Appendix We likewise observe significant difference when comparing the highly educated to the poorly educated user prompt results, and frequently observe that the "poor education" " prefixed tasks have an even better performance than the "high education" ones (average accuracy improvement of 0.1% for "poor educaaiion)). Once again, we fail to replicate the sandbagging phenomenon when utilizing our experimental setup.
3.3 EmotionPromptiing
EmnotionPrompling, presented by Li et al. (2023), augments a lask with emotional cues such as "You'd better be sure" or "This is very important to my career" to enhance problem-solving abilities in LLMs. In the original study, Li et al. (2023) augmented tasks with 11 variations of emotional stimuli and tested six LLMs including ChatGPT and GPT-4. They sourced their tasks from three different benchmark categories, notably using tasks from BIG-Bench (Srivastava et al. 2022). They claim to obtain a "relative performance improvement of 115%% (Li et al. 2023) with their method, arguing that adding emotional component improves the capabilities of LLMs. However, despite the improvement that was strongly implied throughout the original study by raising claims like PinooionPromppt makes it easy to boost the performance of LLMs" Li et al. 2023 the numerical values communicated in the study itself do not coincide with these claims. Instead of communicating the average improvement of the enhanced prompts over the regular prompts, they focused on improvements when cherry-picking the most performant emotional cue. Based on their reported results, we calculated average relative performance improvement of 4.42% BIG-Bench tasks, and a 2.58% relative performance improvement across all benchmarks, when choosing the average performance of all emotional stimuli. Despile identifying this shortcoming in the original study at this early stage, we nevertheless replicated the experiments with our selected tasks and models. We applied the same emotional suffixes as in the original study, apart from "Are you sure?", as LLMs tend to reply to this question, as opposed to solving the given lasks. Similarly to Li el al. (2023)* 's findings, but contrary to their claims, we observed that there was no significant improvement, across every single model and benchmark (see Figure 1 The maximal positive improvement measured is non-significant with an 8.7% difference (x2 = 1.94, - 0.16) (sce Appendix C using Llama 3-8B on CommonsenseQAA. Overall, we observe an insignificant performance increase of 1% when applying FamotionPromptinn (x² = 0.11, p 0.74) (see Appendix C
3.4 Re-Reading
Re- Re-Reading, introduced by Xu et al. 20024) consists in repeating the task verbatim before having the model answer. They compared the baseline performance with the Re-Reading performance, as well as the performance in both conditions when additionally suffixing every task with a chain-of-thoughtt eliciting prompt. The researchers tested GPT-3 (text-davinci-003) (Brown et al. 2020), GPT-3.5, Llama-2-13B and Llama-2-70B (Touvron et al. 2023 in order to have both models with and without instruction fine-tuning. They used a total of 112 arithmetic, common sense, and symbolic reasoning tasks sourced from various datasets with GPT-3 and GPT-3.5, for which they obtained an average gain of 2.7% in accuracy, and 2.9% with the inclusion of chain-of-though reasoning, For Llama 2-13B and Llama 2-70B, they used different set of benchmarks comprising only arithmetic reasoning tasks, with average gain of 2.5% in accuracy (2.7% with chain-of-thoughh reasoning). We replicate the Re-Reading experiments on our selected tasks and models. For this study, we observe a significant improvement for Llama 3-8B (x2 = 13.13,p p < .05) and Llama 3-B0B (x2 = 19.4, p < .05) exclusively (see Appendix C and Figure D. The maximal improvement across all benchmarks for the other models of 2%, for Claude 3 Opus (x" 1.27, p 0.26). Therefore, Re-Reading seems replicable for the Llama 3 models only, which highlights the importance of implementing tests on variety of models. However, the initial study indicated that Re-Reading was effective on GPT
5

---

Published in Transactions on Machine Learning Research (12/2025)
models, notably GPT-3.5, that we also tested with different outcomes. Therefore, only managed to partially replicate the results.
3.5 Rephrase-and- Respood
Beghhaase-nndRResppon inspired by the communication technique of rephrasing to enhance clarity, involves instructing an LLM to first rephrase task and then address it within the same prompt. Deng et al. (2024) demonstrate that simply adding a directive such as "Rephrase and expand the question, and respond" can significantly enhance LLM accuracy. They evaluated GPT-3.5, GPT-4 and Llama 2 across seven different reasoning benchmarks, such as CommonsenseQA and Knowledge Classification (Allen-Zhu & Li 2024), and observed an average improvement of 17.80% across all tested models. We replicated this experiment by incorporating the suggested rephrasing directive to reformulate questions across full dataset. We then applied tailored algorithm to classify the answers, ensuring it could accommodate these new types of responses (sce Appendix B). As for the Re-Reading experiment, we only observe significant positive effect for Llama 3-8B (x² = 10.49, < .05) and Llama 3-70B (x2 18.63, < .05), with the StrategyQA benchmark. Once again, Rephrase-ante and-Respond seems to only show a significant positive improvement on specific benchmark-mode combination, which shows only a partial replication of the results (sce Figure and Appendix
GPT-3.5
GPT-4o
Accur acy (percentage points)
Pro
-3.73
-0.67
Model
Claude Opus
1.07
-1,17
-2.93
-0.27
1.87
-2.27
-0.13
Llama 3-8B
.0.53
2.93
-0.67
0.53
8.27
2.27
6.40
Llama 3-708
-0.27
0.80
4.27
0.00
8.13
5.20
4.00
--15
chain-ot thought thought
Sandbanging High Education
Sandbagging Education
EmattonPrompting
Re Re-RReding
Rephrase- -nd---ad----- Respond
ExpertPrompinng
Prompt Engineering Technique
Figure 1: Accuracy comparisons between the base tests without any prompt modification and the augmented prompls across all LLMss
3.6 ExpertPrompting
ExpertPrompting consists of giving LLMs an instruction to impersonate someone with high expertise on the task subject while completing task. This method presented by Xu et. al. (202 has been greatly

---

Published in Transactions on Machine Learning Research (12/2025)
popularized: it is now recommended in LLM documentations for enhanced LLM accuracy and improved focus adhering to the task's 's requirement. Xu et al. (2025) evaluated the response quality of ExppptPromppiig, assessing aspects like accuracy, helpfulness, or relevance. They evaluated GPT-4 responses with and without ExpertPrompting, which, in the case of the former, possessed a reported higher answer quality 48.5% of the time (Xu et al. 2025). In our experiments, we measure; as with the previous prompl engineering techniques, the accuracy of the ExpertPrompting technique using our set of reasoning benchmarks and L.Mss. We observe no significant improvement across all benchmarks (x² 1.57, p = 0.21) (see Figure [ and Appendix C with
average improvement of only 1% (see Appendix Ch.
4 Identified Issues and Recommendations
Given the identified lack of replicability across various studies, we deem crucial to address the underlying issues contributing to these replication problems in prompt engineering. We have identified specific problems associated with each cited prompt engineering method, which may hint at a broader replication problem in LLM behavior research. In the following, we provide recommendations aimed at mitigating replication problems in prompt engineering and related fields, based our analysis of the selected studies: (1) bench- mark adequacy, (2) model variability and model-benchmark compatibility, and (3) output evaluation and reliability. It is important to note that while an absolute climination of replication problems is unlikely, our recommendations aim to minimize such challenges effectively.
4.1 Benchmark adequacy
4.1.1 Benchmark Quality
In several of the studies examined, we noted major issues regarding the benchmarks used to assess LLM performances. Many benchmark tasks are flawed: they lack proper grammar, contain spelling or punctuation errors (e.g., missing question marks in multiple-choice questions), are semantically nonsensical, incomplete, or factually incorrect. For instance, for CommonsenseeQ (used in the chain-of-thought, Re-Reading, and studies), 10.9% of questions presented punctuation issues easily verifiable with a simple code. Such flaws are not isolated incidents: they reflect a broader pattern of benchmark instability that has been widely discussed in the literature (Gema et al.] 2024 Goetze & Abramssn 2021). Label errors can destabilize benchmark reliability, and even small proportions of mislabeled test data can invert model rankings: phenomenon they call the implies that benchmark noise may disproportionately penalize higher-capacityy models, creating misleading impressions of their reasoning ability. Similarly, researchers have revealed significant ground-truth errors in the Massive Multitask Language Understanding (MMLU) benchmark, one of the most widely used evaluation suites for LLMs (Gema et al.) 2024]. Their re-annotated version, MMLU- Redux, shows that previonsly reported model hierarchies oflen vanish once Lest data are corrected. Likewise, prior work has exposed flaws in widely used reasoning benchmarks, revealing that evaluation setups often reward superficial lexical alignment rather than genuine reasoning (Mousavi et al.) 2025). Together, these findings align with our own observations: large benchmark datasets frequently contain nonsensical, ambiguous, or incorrect items that can distort results.
Rephrase-and-Responn
paradox (Northeutt et al.) 2021 This effect
To mitigate benchmark-induced replication failures, we recommend that researchers rigorously validate benchmark datasets before use. Automated scripts can detect obvious errors (e.g., typos, missing punc- tuation, malformed options), but human verification remains essential for identifying semantic inconsisten- cies. Following research recommendations (Northeutt et al.) 2021 Cao et al. 2025), benchmark curation should be treated as core methodological step rather than auxiliary task. We recommend mention- ing the cleaning procedure and error rates, and encourage the adoption of transparent guidelines such How2Bench (Cao et al., 2025). Where possible, studies should prioritize quality over quantity, using smaller but validated subsets rather than relying on noisy large-scale test sets. This approach minimizes variance, prevents benchmark- we advocate for the publication of "benchmark versioning" and reproducíble evaluation code, so that future research can casily verify whether results stem from model performance or from artifacts in the evaluation data.
illusions of improvement, and enhances comparability across studies. Finally,
7

---

Published in Transactions on Machine Learning Research (12/2025)
4.1.2 Benchmark Coherence and Cross-Study Comparability
A second major source of replication instability arises from inconsistent benchmark selection within and across studies. In several prompt-enginnering papers, models are evaluated on distinct benchmark sets within the same experiment. For example, in the Re- Re-Reading study (Xu et al. 2024), GPT models were tested three reasoning benchmarks, whereas Llama 2 models were evaluated on only onc. In the sandbagging study Perez et al., 2022), researchers evaluate differences in model accuracy when answering questions on the TruthfulQA dataset (Lin et al., 2022). which measures whether a language model is truthful in generating answers to questions, 80 whether the facts mentioned in the answer are correct rather than assessing whether they answered the task correctly. Such inconsistency complicates direct comparison of model performance and raises questions about the rationale behind benchmark choices, potentially leading to benchmark cherry- picking that favors results supporting given hypothesis. Beyond within-study inconsistencies, there is little coherence in benchmark use across related works. In our replication, we applied the same benchmark suite across all experiments to ensure comparability. In contrast, the six studies we examined drew on widely different datasets, some domain-restricted, others deterministic in nature. For instance, the chain- of-thought study evaluated reasoning-centrr tasks, while the EmotionPrommtingg paper used tasks from Instruction Induction (Honovich et al. 2022) and BIG-Bench (Srivastava et al., (022),, such as rhyming or pluralization, which makes results hard to compare as these measure different underlying abilities. Similarly, variations within the process of administering benchmark tasks (notably zero-shot prompting versus few- shot prompting) impact the reasoning process of LLMs and therefore the outcomes. These issues mirror findings by rescarchers who conducted a large-scale analysis of benchmark agreement and showed that minor methodological differences, such as prompt format, evaluation metric, or dataset choice, can drastically alter perceived model performance (Perlitz et al.] 2024). Finally, it is essential to select benchmarks that are conceptually and structurally aligned with the research question. Indeed, when testing prompt techniques claiming alter reasoning performance, given or implied response instructions in the tasks can interfere with output accuracy by restricting the response length and therefore its ability generale more detailed responses, for instance in multiple-choice settings.
Table 1: Benchmark-related replication issues, mitigation techniques, and tools
Issue
Mitigation
Tools
Flawed benchmark tasks: may inflate or deflate perceived performance and favor some models
Include benchmark curation as a core hodollgogaa step
Automated validation scripts
Guidelines such as How2Bench
Prioritize quality over quantity
Cao et al. 2025
Report quality-chock methods and error rates
Inconsistent benchmark selection: complicates cross-study comparison
Report exact benchmarks and what they measure
BenchBench framework PPrlitz
ct al. 2024
Adopt standardized benchmark-agroemennt protocols
Benchmark agreement protocol
Benchmark- obbjectvv misalignment: may distort perceived performance by assessing unrelated capabilities
Ensure coherence between benchmarks and research objectives
BenchBench framework PPeliiz
ct al.
2024)
Benchmark agreement protocol
Document rationale for benchmark choice
8

---

Published in Transactions on Machine Learning Research (12/2025)
We recommend preferring standardized benchmarks across studies in the same field, to ensure that bench- marks are closely aligned with the research objectives. We would advise researchers to ensure that benchmark domains match the specific research objective. We further recommend adopting benchmark-agreemenn proto- cols such as those implemented in BenchBench (Perlitz et al. 2024), which promote standardized evaluation and transparent documentation of benchmark configurations. Establishing and reporting such consistency will reduce variance, mitigate cherry-picking concerns, and make replication efforts more meaningful.
4.2 Model Variability and
Compatibility
4.2.1
Model Variability
In some cases such as in the chain-of-thought prompting study Kojima et al. 2022), the lack of replicability is linked to the models' type and generation. Our results reveal a divergence between GPT-3.5 and GPT- 4o: the former shows mild improvement with chain-of-thougle prompting, whereas the latter displays no significant change. Although neither result. reaches statistical significance, this difference suggests that for earlier-generation models (e.g., GPT-3), used in the replicated study along with other models of that generation, chain-of-thought prompting might have successfully improved LLM accuracy. This aligus with the system cards for recent models, which explicitly warn that techniques like thain-of-thougge reasoning may not improve performance and can even impair it, advising caution in their use with these models (OpenAl 2024). Similarly, we observed a significant improvement for some benchmarks with the Re-Reading and Rephrase-and -espodd prompts, for the Llama 3 models exclusively; if we look at the other models separately, the results are vastly non-significant (see Appendix
C This reinforces the claim that similar experiments may have a considcrably different impact depending on the models usod. Morcover, even when conducting replication experiments using the same models as in the original study, the opacity surrounding model updates and developer prompt modifications in terms of date and type of update (Chen et al. 2024a) renders study replications difficult. Furthermore, the efficacy of prompt engineering depends strongly on the model class; whether the model is inherently reasoning- oriented or relies on surface-level pattern completion. This dynamic is consistent with prior work demonstrating that prompt engineering substantially benefits non-reasoning LLMs but provides marginal or inconsistent improvements for reasoning-tunee ones such as ol-mini (Wang et al. 2024b).
Therefore, it is essential to use variety of models when testing hypothesis, or to at least mention the limited scope of the study when fewer models are used, as effort to prevent a generalization that may be incorrect. In addition to that, some earlier-generation models may not be well-suited for such question- answering experiments requiring reasoning abilities. Specifically, we tested Vicuna 13B and BLOOM on our selected tasks (Appendix A but found the results difficult to interpret, as the accuracy for both the base questions and the prompl-engineered questions was too low to draw reliable insights into the models' behavior. The studies on EmotionPrompttinn BLOOM nonetheless. We recommend consistently clarifying the methodology used for each model, as we were unable to understand how these studies achieved viable results with these models. We recommend that future studies:
Rephrase-ance and-nnnpoon and ExpertPrompting used Vicuna or
1. Explicitly classify models by reasoning capacity when reporting results, distinguishing instruction- following models from reasoning-optimizeed ones.
2. Document model version and release date to ensure comparability over time, as minor provider-side updates can significantly alter performance.
Clarifying these methodological details will help disentangle true performance differences from artifacts introduced by model selection, and will prevent overgeneralization from results obtained on older, non- reasoning LLMs to newer architectures that already internalize these behaviors.
4.2.2 Model-benchmark Compatibility
As large language models evolve, benchmark saturation and data contamination have become central chal- lenges in evaluating prompt-engineering methods. As newer models improve in baseline reasoning perfor-
9

---

Published in Transactions on Machine Learning Research (12/2025)
mance, many existing benchmarks become effectively saturated, leaving little room for prompt engineering to demonstrate measurable gains. For example, tasks that once challenged early-gemeration models are of- ten solved at near-ceiling accuracy by later models, as we can see with the CRT benchmark (Hagendorff et al.) 2023). This saturation reduces statistical sensitivity and can obscure subtle effects introduced by prompling techniques. Tt. is therefore essential to continuously recalibrate benchmark difficulty to match evolving model capabilities. Similarly, many zero-shot and few-shot benchmarks suffer from task Icakage (Li & Flanigan 2024 Wu et al.) 2025 Zhanng et al., 2025); models perform substantially better on datasets predating their training cutoff than on unseen ones, implying widespread contamination. These findings highlight that contamination can make both baseline and prompt-engineered performances appear inflated, thereby diminishing the apparent impact of prompting interventions.
Leakage-free benchmarks such as RandomCalculation (Wu et al. 2025) can be used to isolate genuine rea- soning improvements. Likewise, researchers introduce a reasoning-driver synthesis framework that generates new rescarch-level QA pairs automatically verified for correctness (Zhang et al.) 2025). This method is effec- live for mitigating contamination and memorization effects. Moreover, we recommend using canary strings when publishing benchmark data to help detect and quantify potential data leakage in future models trained on that data. By embodding unique, traccable strings within benchmark datasets, rescarchers can later probe models to see whether these sequences are reproduced, thus revealing unintended memorization. This proac- tive approach further enhances the reliability of evaluation results. Finally, prompt effectiveness depends on both model architecture and task characteristics (Chen et al. 2024b). For an optimal performance, different prompts should be adapted to different LLMs, which may explain some differences observed in this study, especially when the original prompt engineering studies were applied a small sample of LLMss. A "one- sizc-fits-all" benchmark cvaluation approach can longer capture the true impact of prompt-ongineering methods, without comparing the exact same models. Other than expanding the number and variety of LLMs tested, a possible solution would be to adapt these prompt engineering techniques to each LLM using model-adaptive prompt optimizers (Chen et al., 2024bb to calibrate the prompts to the models.
We recommend that future studies:
Adjust benchmark difficulty to prevent saturation: when newer reasoning models reach near-ceiling performance, evaluations should shift loward more complex or reasoning-intensive lasks.
2. Mitigate contamination by verifying dataset release dates and removing items that predate model training cutoffs.
3. Employ model-adaplive evaluation strategies that align prompts and benchmarks with each model's capabilities in question, reducing misleading cross-model comparisons.
4. Report benchmark provenance and contamination checks as part of all experimental documentation, alongside model version and date, to promote transparency and interpretability.
Broaden model coverage to strengthen the generalizability of findings.
Ultimately, as LLMs integrate reasoning processes internally, older benchmarks cease to serve as meaningful discriminators of model capability or prompt-engineering effectiveness. Reliable evaluation now requires not only harder benchmarks such Humanity's Last Exam (Phan et al., 2025), but also methodologies that explicitly account for monel-benchmark interactions: how specific benchmarks align with, advantage, contaminate particular models, thereby shaping observed results.
4.3
Output Evaluation and Reliability
4.3.1
Evaluation Reliability
For evaluations of LLMs, it is key to ensure the accuracy of the LLM output classifications. We have attempted to replicate a large number of verification techniques presented in other studics. However, when checking the accuracy of these techniques, we discovered that a significant number of them had shortcomings. For example, functions based solely Regex rules were generally too vague, leading to flawed classifications.
10

---

Published in Transactions on Machine Learning Research (12/2025)
Table 2: Model-related replication issues, mitigation techniques, and tools
Issue
Mitigation
Relevant Tools
Model variability and opacity: performance differences across model generations and families
Classify models by reasoning capacity (reasoning vs.
non-reasoning)
Document exact model version
Benchmark saturation and contamination: newer models reach near-ceiling accuracy or show inflated results for pre-cutoff datasets
Recalibrate benchmark difficulty using reasoning-intensive
Leakage-free dalasets such as CaladomCleeultion (Wu ct al.
intensive tasks
Verify dataset release dates against model training cutoffs
2025
Beyond Memorization
synthesis framework (Zhagg
Use leakage-free dalasets
et al. 2025
Check for canary strings
Harder benchmarks such as
Humanity's 's Last Exam
Phan et al. 2025
Model-benchmmrr incompatibility: some benchmarks advantage or disadvantage certain model types, skewing generalization
Conduct cross-model evaluat ions on identical benchmarks
Other metrics, such the F1 word overlap score, do not work effectively, as they would classify correct LLM outputs incorrect, simply because the token length differed too much from the ground truth. Similar issues have been found with metrics like BLEU, ROUGE METEOR (Ng & Abrecht, 2015 Liu et al. 2017)
Moreover, studies often rely using LLM-as-a-judge to classily LLM outputs (Pan et al. 2023). such the ExpertPromptinn study (Li et al. 2023). Exclusively relying on LLM-based evaluation can introduce flaws. When applying the original or even enhanced versions of their LLM output classification prompts, we observed multiple issues that compromised evaluation reliability. For instance, the judge would produce inconsistent evaluations when asked multiple times to assess the same output. Tt was also influenced by the task when classifying responses, and the verification prompt needed to be task-specifie and highly precise. Prior research has also documented LLM-as-a-judge biases that may further distort classifications, such the self-enhancement bias, lendency for an L.L judge to rate outputs from its own model family more favorably, or the verbosity bias, where longer responses are rated more positively ęZheng et al. 2023 As some prompting techniques, such as choin-of-thought prompting, systematically increase response length (Appendix D), verbosity bias may artificially inflate measured accuracy when using LLM judges. Given the time-consuming aspect of manually double-checking classifications, we also suspect that this done very rarely. Furthermore, among the papers we replicated, one did not include any details about the verification process (Li et al. 2023); it goes without saying that any verification process should be clearly reported in each study, to make the study replication feasible. We therefore recommend developing more precise and task-spocific verification methods, and ensuring thorough documentation of these processes in all studies to facilitate accurate replication and validation of results. We recommend avoiding reliance solely on LLM- based evaluation; if alternative evaluation strategies are not feasible, then we would recommend running multiple iterations per prompt and aggrogating the results mitigate potential misjudgments. We also recommend that creators of new benchmarks provide standardized verification process, encouraging all users to apply the same verification criteria.
11

---

Published in Transactions on Machine Learning Research (12/2025)
4.3.2 Evaluation Metrics
The choice of evaluation metric is as critical as the verification method itself. Different metrics capture fundamentally distinct dimensions of model behavior, and results are not directly comparable across them. For instance, accuracy reflects factual correctness, whereas output quality captures fluency, coherence, or stylistic adoquacy. Similarly, pairwise comparison approaches where an LLM or human judge selects the "belter" of two outputs- measure relative preference, not absolute performance. Treating these hetero- geneous measures as equivalent can therefore produce misleading conclusions about the effectiveness of a prompting technique. For example, in our study, the original Prompting study focused on output quality rather than accuracy. We replicated their comparative evaluation approach using our own bench- marks. To ensure an unrestricted quality assessment, we removed formatting instructions from our prompts and evaluated complete LLM outputs. Our results showed higher quality scores for ExpertPrompting over base prompts, with winning percentages ranging from 26.53% (x² == 82.65, p .05) for Gemini 1.5 Pro, to 76.93% (x2 294.27,, < .05) for Llama 3 8-7 (see Figure 2 Despite these significant results, we re- main skeptical about the robustness of this evaluation method. Relying solely on an LLM to assess outputs from another introduces risk of misjudgment, as seen in Section 4.3.1 Moreover, this pairwise evaluation method frequently selected a "winner" even when both responses were factually incorrect. In some cases, the model even favored an inaccurate response while penalizing an accurate one. Based on this observation, we re-evaluated the same responses for factual accuracy using a tailored version of our accuracy assessment. Once again, no significant. improvement was observed. In fact, ExpertPrompting led to decreased accuracy, ranging from drop of 2.4% (x2 0.84, p 0.36) for Gemini 1.5 Pro, to 14.53% (x2 34.00, < .05) for Llama 3-8B. This raises an important question: if response is rated as "high quality" despite being factually incorrect, can it truly be considered an improvement? This can be misleading for users, who might reasonably assume that accuracy is embedded within the "quality" metric. Consequently, despite some stud- ies presenting their claims similarly using verbs such as "improves", "enhances", "overperforms" to describe their prompting techniques, their outcomes cannot effectively be compared. Including both quality and ac- curacy metrics in evaluations would have provided more comprehensive understanding of the effectiveness of, c.g., ExpertPromptinn.
We would recommend including at least one factual metric when using reasoning benchmarks. Combining complementary metrics would also provide a more comprehensive picture of prompt effectiveness and guards against overinterpretation of singlo-score gains.
4.3.3 Reporting and Comparing Results Transparently
Across the replicated prompt-engineering studies, we observed substantial methodological variation that complicates the interpretation and comparison of results. Some studies notably display a particularly poor or unclear scientific method. In the AinotionPrompling study (Li et al. 2023). researchers cherry-pick the prompt with the best result out of eleven different prompts, rather than calculating an average across all prompts. This seemingly deliberate action may be due to a publication bias, which motivates rescarchers to manipulate results to be positive and therefore publishable. In addition, some studies, such as the Re-Reading study (Xu et al. 2024), report results as "significant" " multiple times without presenting the corresponding statistical calculations
p-values. This lack of statistical transparency can mislead readers into assuming statistical significance without the necessary evidence to support such claims. It is crucial that when terms like "significant" " are used, they are backed by clearly defined statistical measures. Moreover, some studies do not properly report the details of their experimental setup (Perez et al. 2022), which makes it confusing or even impossible to understand and therefore to replicate their process exactly. In this case, the lack of transparency forbids us from detecting possible shortcomings. We recommend adopting standardized evaluation methodologies and elearly defining metries to ensure that results from different studies can be accurately compared and interpreted.
12

---

Published in Transactions on Machine Learning Research (12/2025)
Comparison Answer Quality (ExpertPrompting vs Base)
Accuracy
Prompt Type
Performance of ExpertPromptiin answers Vs Base answers
LOSE
LOSE 4.5%
LOSE 8. 7%%
Base
11.1%
LOSE 17. 17.9%
LOSE 17.37 3%
Expert
TIE
0%
30.4% 4%
TIE
41 7%
TIE 67.0%
Proportion Correct
55.
40
WIN
40
WIN 7%
76.99
WIN
5%
WIN
20
45 5%
20
VIN
26 5%
GPT-3.5
GPT40
Gemini 1.5 Pro
Claude Opus
Liama 88
Uama 3-708
GPT-3.5
GPT-40
Claude Opus
Uaaaa +888
Uama 3-708
Gemini 1.5
Figure 2: Comparison of accuracy and quality between the base tests and without formatting constraints, across all LLMs. Error bars show 95% Cls.
using prompts
Table 3: Results-related replication issues and mitigation techniques
Issue
Mitigation
Evaluation unreliability: unstable or imprecise LLM-as-a-judge scoring
Use precise, task-specific verification prompts
Run multiple judging prompts and models and average across results
Include manual or based cross-chocks
Metric inconsistency: mixing incomparable measures (e.g., accuracy vs. quality)
Define all metrics clearly and report how each is computed
Include at least one factual accuracy metric in reasoning tasks
Cherry-picking and poor reporting:
Aggregate across all tested prompts justify subset selection Report all statistical tests, p-values, and confidence intervals
selective results, missing p-values, unclear methodology
Provide full methodological details (benchmarks, models, and prompts)
13

---

Published in Transactions on Machine Learning Research (12/2025)
5 Transparency Recommendations and Modular Evaluation Pipeline
To ensure methodological transparency and facilitate replication, future studies should explicitly document the following:
Benchmarks
Exact benchmark names (and versions if applicable)
Clear description of what each benchmark measures
Quality-check methodology, applied (e.g,, automated validation scripts, manual review)
Dataset provenance, including any filtering, sampling, or modification procedures
Models
Exact model names and versions
Description of model class (reasoning non-reasoning)
Full inference methodology, e.g., temperature, max tokens, random seeds
used or modified, system prompts and/or added prompt instructions
Evaluation and Metrics
Exact metrics reported (e.g., accuracy, quality, interpretability) and how they were computed
If using LLM-as-a-judge: specify the judging model, version, full evaluation prompt, and aggregation method if any
All statistical analyses: e.g., test type, p-values, confidence intervals
To support transparency and facilitate replication, we developed a modular evaluation pipeline that accom- panies this study. The code was designed to allow researchers to replicate, extend, or adapt our experiments with minimal effort. Each stage of the evaluation process is modularized, enabling flexible modification of:
Benchmark selection: users can casily substitute or add benchmarks, specify the number of random items drawn, and control sampling strategies.
Prompt-engineering techniques: zero-shot prompting methods, particularly those relying on prefix or suflix additions, can be switched or customized through dedicated modules.
Model configuration: the model used for inference can be easily changed, facilitating cross-model comparisons.
Evaluation and verification setup: the output classification component allows users to define or modily the verification prompt and evaluate output quality or accuracy according to their own criteria.
This modular design serves dual purpose. First, it allows rapid, small-scale experiments to obtain pre- liminary estimate of the replicability of prior findings before committing to large-scale runs. Second, it encourages exploration and calibration: users can quickly test which combination of model, benchmark, prompting strategy, and evaluation prompt is most appropriate for their research context. The code is accessible here: https://github. http:////aaaaeeeeeee/eeeeataaaea
By lowering the practical barrier replication and promoting iterative experimentation, we aim to foster culture of methodological trans- parency and empirical verification in prompt-engineering research.
14

---

Published in Transactions on Machine Learning Research (12/2025)
6 Discussion
Our experiments show that most of the tested prompt engincering techniques do not lead replicable generalizable performance improvements in LLMs. Most techniques, when applied in slightly different experimental setups, failed to produce the claimed benefits. Some techniques occasionally even resulted in a decrease in response accuracy. In view of the uncritical propagation of the prompt enginocring tochniques in the literature (Schulhoff et al. 2024), we recommend a more cautious approach when citing papers with insufficient methodological standards. While further research could help reliably understand the conditions under which specific techniques are effective, we offer recommendations and tools mitigate these replication issues.
While our focus is replication issues in prompt engineering, it might be the case that similar challenges exist in other LLM behavior evaluation techniques. Our findings suggest that further research is necessary to reliably understand the conditions under which specific prompt engineering techniques are effective, and they underscore the need for replication studies across the broader spectrum machine behavior research. Only by rigorously verifying or refuting insights from these studics can we build a more reliable foundation for evaluating LLM performance.
In line with the recommendations proposed above, emphasize the importance of enhanced research trans- parency and the application of rigorous scientific methods when evaluating LLM behavior. Future investi- gations should not only continue to scrutinize prompt engineering approaches but also exlend replication efforts to other fields of LLM evaluations to ensure the robustness and generalizability of findings LLM behavior research.
Resource availability
The datasets and code generated during this study are available in the "RepicationCCrissnIILAMMaaa uation' repository on the Open Science Framework (OSF) at http:://ooff.oooccgg////weee onlyy
FB2/231117744f88877774446666 and at http:////gttttbb.c//////eu/eeeee http://tt...cc.aaarrree////a/aaa
Declaration of interests
The authors declare competing interests.
References
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.2, knowledge manipulation. arXiv,
2024. doi: (0.48550/arXiv.222 230..340... URL htttps://di..rg//11 48550/ arxiv. wwx.i..309....... 14402
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané, Concrete problems in ai safety. arXiu, 2016. doi: T0.45550/axXiw.1166.66666
URL http::::/a.i...rr//......
arXiv.1606.066.
Anthropic. Claude model card. https:///ocs. http:///os.aatoolcccccccc///c//s 2024.
Accessed: 2024-09-06.
Marcel Binz and Eric Schulz, Using cognitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences, (20(6))c2218231111 2023. doi: 10.1073 00073/pnss.22 221..2211523122 URL https:///ot.orrr 10. 10.107//pna.. 2218523120
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krucger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Nya Sutskever, and Dario
15

---

Published in Transactions on Machine Learning Research (12/2025)
Amodei. Language models are few-shot learners. arXiv, 2020. doi:
URL
https://doi.ors /tg//.. 20050//rXiv..o 2005. 14165
Jialun Cao, Yik-Kit Chan, Zixuan Ling, Wenxuan Wang, Shuqing Li, Mingwei Liu, Ruixi Qiao, Yuting Han, Chaozheng Wang, Boxi Yu, Pinjia He, Shuai Wang, Zibin Zheng, Michael R. Lyu, and Shing- Chi Cheung. How should we build a benchmark? revisiting 274 code-related benchmarks for Ils,, 2025. URL https://arxiv.. www.w//.rggbbaaaaeaa aatpaa////2001.00 10711
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. A survey on evaluation of large language models. arXiv, 2023. doi: 10.48550/arXiv.23077 https://doi. org/ 10. www.448550/21113 2307.03109
URL
Lingjaa Chen, Matei Zaharia, and James Zou. How Is ChatGPT's Behavior Changing Over Time? Harvard Data Science Review, 6(2), mar 12 2024a. https:/ http://dssrmmm. www.teeeeemtttedeuu.m httph//pbbbyy/ym.....
Yuyan Chen, Zhihao Wen, Ge Fan, Zhengyu Chen, Wei Wu, Dayiheng Liu, Zhixu Li, Bang Liu, and Yanghua Xiao. Mapo: Boosting large language model performance with model-adaptive prompt optimization, 04118
2024b. URL https://arxiv org/
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna; An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URI.
2023-03-30-vicuna
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXie, November 2021. doi: 10.48550 10.48550/arXiv2211...a 21111/111....11116...... URL
ottp::///oii.or/11..4.888. 48550/arXiv 2110. 14168
Yihe Dong, Weitong Zhang, Zixiang Chen, and Quanquan Gu. Rephrase and respond: Let large language models ask better questions for themselves. Xiv, 2024. doi: 10.48550/ar (118555//rXii.2311..000 URL
https:
//doi.org/10 httpa//rr/.1...55//rra.....
2311 2311.04205
Mingqi Gao, Xinyu Hu, Jie Ruan, Xiao Pu, and Xiaojun Wan. Llm-based nlg evaluation: Current status and challenges. arXiu, 2025. doi: 10.48550/arXiv http//arxxvv.22.112222 2402.01383 URL https::/doi. 48550//x11x/ 2402.01388
Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile van Krieken, and Pasquale Minervini. Are we done with milu?? arxiv, 2024. URL http:://aaii:: www.:/org/bb/aaeeeeo 04127 Version 1.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle laptop? question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346 361, 2021.
Trystan S. Goetze and Darren Abramson. Bigger isn't better: The ethical and scientific vices of extra-large datasots in language modcls. In Companion Publication of the 13th ACM Web Science Conference 2021, 34466809. URL http::://///// wrrpla..rr/o... 0e1111114434444 3446667311.166199 3466809
pp. 69 75, 2021. doi:
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. Xrv, 2024. doi: (88550/arXiv.242..2
URL https://doi.orgg 1tte/10.4855//rxiiiii
Thilo Hagendorff. Mapping the ethics of generative ai: A comprehensive scoping review. Minds and Machines, 34(4):39, 2024. doi: 02.1077//10022-00000944. URL htttp::/ddo.oor/1o..rr org/10.
611023-024-09694-
16

---

Published in Transactions on Machine Learning Research (12/2025)
Thilo Hagendorff, Sarah Fabi, and Michal Kosinski. Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in chatgpt. Nature Compulational Science, 3(10)8333- 838, 2023. doi: (4.1008//443588022--00000
URL
Susan Hao, Piyush Kumar, Sarah Laszlo, Shivani Poddar, Bhaktipriya Radharapu, and Renee Shelby, Safety and fairness for content moderation in generative models. arXiv, 2023. doi: 11148550/arXiv.3366... URL https://doi org/10 10. 48550/855Xxi 2306. 2306.06135
Or Honovich, Uri Shaham, Samuel R. Bowman, and Omer Levy. Instruction induction: From few examples natural language task descriptions. arXiv, 2022. doi: 10.48550 2at:://aaaiiv.22.11111 URL https://doi,
org/10. 10. 48550/ arXiv www...www..............a.......e 2205. 10782
Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Lukas Vierling, Donghai Hong, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng, Juntao Dai, Xuchai Pan, Kwan Yee Ng, Aidan O'Gara, Hua Xu, Brian Tse, Jie Fu, Stephen McAleer, Yaodong Yang, Yizhou Wang, Song-Chun Zhu, Yike Guo, and Wen Gao. Ai alignment: A comprehensive survey. arXiv, 2024. URL https://doi. 1ttp///////....///rrii..... 48550/arXiv.
doi: 10.48550/arxii
2310.19852
Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, and Mengnan Du. The impact of reasoning step length on large language models. arXiv, 2024. URL http://arxiy) org/ http//bb//41.0555
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Twasawa. Large language models are zero-shot reasoners. Xiv, 2022. URL htttps://aarrvv.or//bb////000000 11916
Changmao Li and Jeffrey Flanigan. Task contamination: Language models may not be few-shot anymore. Proceedings of the AAAI Conference on Artificial Intelligence, 38(16):1847111888) Mar. 2024. doi: 10. 1609/ (009//aaivv/ http//aaavvvvvll./ v38116.29808 URL https://ojs wwt.:aaaa.....naa.eeaaa wrw//ooer.//neooooooooooooog.ege /htp://AAI/aarricc/////////aa
Cheng Li, Jindong Wang, Yixuan Zhang, Kaijic Zhu, Wenxin Hou, Jianxun Lian, Fang Luo, Qiang Yang, and Xing Xie. Emotionprompt: Leveraging psychology for large language models enhancement via emotional stimulus, 2023. URL
wttp://wwww.rggba////0000 11760
Chin-Yer Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https: //aclanthology. org///410/// /acllanttologyyoorr//00-111/3
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214-3252, 2022.
Chia-We Liu, Ryan Lowe, Iulian V. Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation, 2017. URL https:://arxiv.or//bbb.s..
wwwgg//baa/10g// 08023
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig, Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2021. doi: 00.1145/356083 URL https:::::// httpps://doi org/10. wtt....../////............o..... 1145/35608885
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Proceedings of the 36th International Conference on Newral Information Processing Systems, pp. 2507 2521, 2022.
Shervin Minace, Tomas Mikolov, Narjes Nikzad, Mcysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey. arXiv, 2024. doi: 48555//araii.440/0/ 240.002.00.00116.. URL htaps:://aoi.org/1....44 000.8850000022222220000000000
2402.06196
17

---

Published in Transactions on Machine Learning Research (12/2025)
Swaroop Mishra, Arindam Mitra, Necraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, and Ashwin Kalyan. Numglue: A suite of fundamental yet challenging mathematical reasoning tasks. Tn Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5503-3523. 2022.
Seyed Mahed Mousavi, Edoardo Coechinato, Lucia Hornikova, and Giuseppe Riccardi. Garbage in, reasoning out? why benchmark scores are unreliable and what to do about it, 2025. URL https: org/
abs/2506. 23864
Marianna Nezhurina, Lucia Cipolina-Kun, Mchdi Cherti, and Jenia Jitsev. Alice in wonderland: Simple tasks showing complete reasoning breakdown in state-of-the-art large language models, 2024. URL. https:
//arxiv..o wwt. 2atpbb/2460.00000 02061
Jun- Lin-Ping Ng and Viktoria Abrecht. Better summarization evaluation with word embeddings for rouge, 2015.
URL httpps///aaxivvogg/bbbb///////// 11001100880000444
Curtis G. Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks, 2021. URL
OpenAT. Introducing chatgpt, November 2022. URL. http:://peeaa..co/mmeeeeedtaaaa
OpenAL. Gpt-4 technical report, 2023. URL httpppps//dhh..pppaaaca.capaaapp
OpenAL. Openai platform, 2024. 2024-09-13.
http://wwww....oocooooo//aaaaaaa Accessed:
Alexander Pan, Jun Shern Chan, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Jonathan Ng, Hanlin Zhang, Scott Emmons, and Dan Hendrycks. Do the rewards justify the means? measuring trade- offs between rewards and ethical behavior in the machiavelli benchmark, 2023. /ttpsss/////r
abs/2304.0323
Roger Pong. Reproducible research in computational science. Science (New York, N. N.Y.J. 334:1226-1227, 2011. doi: 10.1126/scieess 1213847 URL https://doi. http///0..12///ceeeeeee. 1213847
Ethan Perez, Sam Ringer, Kamilè Lukošiūté, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Is- rael, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jarod Mueller, Jecyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Lianc Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemí Mer- cado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfeed-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Fvan Hubinger, Nicholas Schiefer, and Jared Kaplan. Discovering language model behaviors with model-written evaluations. arXiv, 2022. Ctt:htt::
Yotam Perlitz, Ariel Gera, Ofir Arviv, Asaf Yehudai, Elron Bandel, Eyal Shnarch, Michal Shmueli-Scheuer and Leshem Choshen. Do these 11m benchmarks agree? fixing benchmark evaluation with benchbench, 2024. URL https://arxiv org/ abs/2407 13696
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mo- hamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, Dmitry Dodonov, Tung Nguyen, Jacho Lee, Daron Anderson, Mikhail Doroshenko, Alun Cennyth Stokes, Mobeen Mahmood, Oleksandr Pokutnyi, Oleg Iskra, Jessica P. Wang, John-Clark Levin, Mstyslav Kazakov, Fiona Feng, Steven Y. Feng, Haoran Zhao, Michael Yu, Varun Gangal, Chelsea Zou, Zihan Wang, Serguei Popov, Robert Gerbicz, Geoff Galgon, Johannes Schmitt, Will Ycadon, Yongki Lee, Scott Sauers, Alvaro Sanchez, Fabian Giska, Marc Roth, Soren Riis, Saiteja Utpala, Noah Burns, Gashaw M. Goshu, Mohinder Maheshbhai Naiya, Chi- dozie Agu, Zachary Giboney, Antrell Cheatom, Francesco Fournier-Facio, Sarah-Jane Crowson, Lennart
18

---

Published in Transactions on Machine Learning Research (12/2025)
Finke, Zerui Cheng, Jennifer Zampese, Ryan G. Hoerr, Mark Nandor, Hyunwoo Park, Tim Gehrunger, Jiaqi Cai, Ben McCarty, Alexis C Garretson, Edwin Taylor, Damien Sileo, Qiuyu Ren, Usman Qazi, Lianghui Li, Jungbae Nam, John B. Wydallis, Pavel Arkhipov, Jack Wei Lun Shi, Aras Bacho, Chris G. Willcocks, Hangrui Cao, Sumeet Motwani, Emily de Oliveira Santos, Johannes Veith, Edward Vendrow, Doru Cojoc, Kengo Zenitani, Joshua Robinson, Longke Tang, Yugi Li, Joshua Vendrow, Natanael Wild- ner Fraga, Vladyslav Kuchkin, Andrey Pupasov Maksimov, Pierre Marion, Denis Efremov, Jayson Lynch, Kaiqu Liang, Aleksandar Mikov, Andrew Gritsevskiy, Julien Guillod, Gözdenur Demir, Dakotah Mar- tinez, Ben Pageler, Kevin Zhou, Saeed Soori, Ori Press, Henry Tang, Paolo Rissone, Sean R. Green, Lina Brüssel, Moon Twayana, Aymeric Dieuleveut, Joseph Marvin Imperial, Ameya Prabhu, Jinzhou Yang, Nick Crispino, Arun Rao, Dimitri Zvonkine, Gabriel Loiseau, Mikhail Kalinin, Marco Lukas, Ciprian Manolescu, Nate Stambaugh, Subrata Mishra, Tad Hogg, Carlo Bosio, Brian P Coppola, Julian Salazar, Jaehyeok Jin, Rafael Sayous, Stefan Ivanov, Philippe Schwaller, Shaipranesh Senthilkuma, Andres M Bran, Andres Algaba, Kelsey Van den Houte, Lynn Van Der Sypt, Brocht. Verbeken, David Noever, Alexci Kopylov, Benjamin Myklebust, Bikun Li, Lisa Schut, Evgenii Zhellonnzhkkii, Qiaochu Yuan, Derek Lim, Richard Stanley, Tong Yang, John Maar, Julian Wykowski, Martí Oller, Anmol Sahu, Cesare Giulio Ardito, Yuzheng Hu, Ariel Ghislain Kemogne Kamdoun, Alvin Jin, Tobias Garcia Vilchis, Yuexuan Zu, Martin Lackner, James Koppel, Gongbo Sun, Daniil S. Antonenko, Steffi Chern, Bingchen Zhao, Pierrot Arsene, Joseph M Cavanagh, Daofeng Li, Jiawei Shen, Donato Crisostomi, Wenjin Zhang, Ali Dehghan, Sergey Ivanov, David Perrella, Nurdin Kaparov, Allen Zang, Ilia Sucholutsky, Arina Kharlamova, Daniil Orel, Vladislav Poritski, Shalev Ben- David, Zachary Berger, Parker Whitfill, Michael Foster, Daniel Munro, Linh Ho, Shankar Sivarajan, Dan Bar Hava, Aleksey Kuchkin, David Holmes, Alexandra Rodriguez- Romero, Frank Sommerhage, Anji Zhang, Richard Moat, Keith Schneider, Zakayo Kazibwe, Don Clarke, Dae Hyun Kim, Felipe Meneguitti Dias, Sara Fish, Veit Elser, Tobias Kreiman, Victor Efren Guadar- rama Vilchis, Immo Klose, Uijwala Anantheswaran, Adam Zweiger, Kaivalya Rawal, Jeffery Li, Jeremy Nguyen, Nicolas Daans, Haline Heidinger, Maksim Radionov, Václav Rozhoñ, Vincent Ginis, Christian Stump, Niv Cohen, Rafal Poświata, Josef Tkadlec, Alan Goldfarb, Chenguang Wang. Piotr Padlewski, Stanislaw Barzowski, Kyle Montgomery, Ryan Stendall, Jamie Tucker-Foltz, Jack Stade, T. Ryan Rogers, Tom Goertzen, Declan Grabb, Abhishek Shukla, Alan Givré, John Arnold Ambay, Archan Sen, Muham- mad Fayez Aziz, Mark H Inlow, Hao Hc, Ling Zhang, Younesse Kaddar, Ivar Ängquist, Yanxu Chen, Har- rison K Wang, Kalyan Ramakrishnan, Elliott Thornley, Antonio Terpin, Hailey Schoelkopf, Eric Zheng, Avishy Carmi, Ethan D. L. Brown, Kelin Zhu, Max Bartolo, Richard Wheeler, Martin Stehberger, Peter Bradshaw, JP Heimonen, Kaustubh Sridhar, Ido Akov, Jemnifer Sandlin, Yury Makarychev, Joanna Tam, Hicu Hoang, David M. Cunningham, Vladimir Goryachev, Demosthenes Patramanis, Michael Kranse, An- drew Redenti, David Aldous, Jesyin Lai, Shanmon Coleman, Jiangnan Xu, Sangwon Lee, Thas Magoulas, Sandy Zhao, Ning Tang, Michael K. Cohen, Orr Paradiso, Jan Hendrik Kirchner, Maksym Ovchynníkov, Jason O. Matos, Adithya Shenoy, Michael Wang, Yuzhou Nie, Anna Sutyber-Betley, Paolo Faraboschi, Robin Riblet, Jonathan Crozier, Shiv Halasyamani, Shreyas Verma, Prashant Joshi, Eli Meril, Ziqiao Ma, Jérémy Andréoletti, Raghav Singhal, Jacob Platnick, Volodymyr Nevirkovets, Tuke Basler, Alexan- der Ivanov, Seri Khoury, Nils Gustafsson, Marco Piccardo, Hamid Mostaghimi, Qijia Chen, Virendra Singh, Tran Quoc Khánh, Paul Rosu, Hannah Szlyk, Zachary Brown, Himanshu Narayan, Aline Menezes, Jonathan Roberts, William Alley Kunyang Sun, Arkil Patel, Max Lamparth, Anka Reuel, Linwei Xin, Hanmeng Xu, Jacob Loader, Freddic Martin, Zixuan Wang, Andrea Achilloos, Thomas Preu, Tomek Ko- rbak, Ida Bosio, Fereshteh Kazemi, Ziye Chen, Biró Bálint, Eve J. Y. Lo, Jiaqi Wang, Maria Inês S. Nunes, Jeremiah Milbauer, M Saiful Bari, Zihao Wang, Behzad Ansarinejad, Yewen Sun, Stephane Du- rand, Hossam Elgnainy, Guillaune Douville, Daniel Tordera, George Balabanian, Hew Wolff, Lynna Kvistad, Hsiaoyun Milliron, Ahmad Sakor, Murat Eron, Andrew Favre D. O., Shailesh Shah, Xiaox- iang Zhou, Firuz Kamalov, Sherwin Abdoli, Tim Santens, Shaul Barkan, Allison Tee, Robin Zhang, Alessandro Tomasiello, G. Bruno De Luca, Shi-Zhuo Looi, Vinh-Kha Le, Noam Kolt, Jiayi Pan, Emma Rodman, Jacob Drori, Carl J Fossum, Niklas Muennighoff, Milind Jagota, Ronak Pradeep, Honglu Fan, Jonathan Eicher, Michael Chen, Kushal Thaman, William Merrill, Moritz Firsching, Carter Harris, Ste- fan Ciobâcă, Jason Gross, Rohan Pandey, Πya Gusev, Adam Jones, Shashank Agnihotri, Pavel Zhelnov, Mohammadreza Mofayezi, Alexander Piperski, David K. Zhang, Kostiantyn Dobarskyi, Roman Leven- tov, Ignat Soroko, Joshua Duersch, Vage Taamazyan, Andrew Ho, Wenjie Ma, William Held, Ruicheng Xian, Armel Randy Zebaze, Mohanad Mohamed, Julian Noah Leser, Michelle X Yuan, Laila Yacar, Jo-
19

---

Published in Transactions on Machine Learning Research (12/2025)
hannes Lengler, Katarzyna Olszewska, Claudio Di Fratta, Edson Oliveira, Joseph W. Jackson, Andy Zou, Muthu Chidambaram, Timothy Manik, Hector Haffenden, Dashiell Stander, Ali Dasouqi, Alexander Shen, Bita Golshani, David Stap, Egor Kretov, Mikalai Uzhou, Alina Borisovna Zhidkovskaya, Nick Win- ter, Miguel Orbegozo Rodriguez, Robert Lanff, Dustin Wehr, Colin Tang, Zaki Hossain, Shaun Phillips, Fortuna Samuele, Fredrik Ekström, Angela Hammon, Oam Patel, Faraz Farhidi, George Medley, For- ough Mohammadzadeh, Madellene Peñafoor, Haile Kassahun, Alena Friedrich, Rayner Hernandez Perez, Daniel Pyda, Taom Sakal, Omkar Dhamane, Ali Khajegili Mirabadi, Eric Hallman, Kenchi Okutsu, Mike Battaglia, Mohammad Maghhsudimmehrabani, Alon Amit, Dave Hulbert, Roberto Pereira, Simon Weber, Handoko, Anton Peristyy, Stephen Malina, Mustafa Mchkary, Rami Aly, Frank Reidegeld, Anna-Katham Dick, Cary Friday, Mukhwinder Singh, Hassan Shapourian, Wanyoung Kim, Mariana Costa, Hubeyb Gur- dogan, Harsh Kumar, Chiara Ceconello, Chao Zhuang, Haon Park, Micah Carroll, Andrew R. Tawfeek, Stefan Steinerberger, Daattavya Aggarwal, Michael Kirchhof, Linjie Dai, Evan Kim, Johan Ferret, Jainam Shah, Yuzhou Wang, Minghao Yan, Krzysztof Burdzy, Lixin Zhang, Antonio Franca, Diana T. Pham, Kang Yong Loh, Joshua Robinson, Abram Jackson, Paolo Giordano, Philipp Petersen, Adrian Cosma, Jesus Colino, Colin White, Jacob Votava, Vladimir Vinnikov, Ethan Delaney, Petr Spelda, Vit Stritecky, Syed M. Shahid, LeannChristopher Mourrat, Lavr Vetoshkin, Koen Sponselee, Renas Bacho, Zheng- Xin Yong, Florencia de la Rosa, Nathan Cho, Xiuyu Li, Guillaume Malod, Orion Weller, Guglichmo Albani, Leon Lang, Julien Laurendeau, Dmitry Kazakov, Fatimah Adesanya, Julien Portier, Lawrence Hollom, Victor Souza, Yuchen Anna Zhou, Julien Degorre, Yigit Yalm, Gbenga Daniel Obikoya, Rai, Filippo Bigi, M. C. Boscá, Oleg Shumar, Kaniuar Bacho, Gabriel Recchia, Mara Popescu, Nikita Shulga, Ngefor Mil- dred Tanwie, Thomas C. H. Lux, Ben Rank, Colin Ni, Matthew Brooks, Alesia Yakimchyk, Huanxu, Lüu, Stefano Cavalleri, Olle Häggström, Emil Verkama, Joshua Newbould, Hans Gundlach, Leonor Brito- Santana, Brian Amaro, Vivek Vajipey, Rynaa Grover, Ting Wang, Yosi Kratish, Wen-Ding Li, Sivakanth Gopi, Andrea Caciolai, Christian Schroeder de Witt, Pablo Hernández- Hamaaa, Emanuele Rodolå, Jules Robins, Dominic Williamson, Vincent Cheng, Brad Raynor, Hao Qi, Ben Segev, Jingxuan Fan, Sarah Martinson, Erik Y. Wang, Kaylic Hausknocht, Michael P. Brenner, Mao Mao, Christoph Demian, Peyman Kassani, Xinyu Zhang, David Avagian, Eshawn Jessica Scipio, Alon Ragoler, Justin Tan, Blake Sims, Rebeka Pleeník, Aaron Kirtland, Omer Faruk Bodur, D. P. Shinde, Yan Carlos Leyva Labrador, Zahra Adoul, Mohamed Zekry, Ali Karakoe, Tania C. B. Santos, Samir Shamseldeen, Loukmane Karim, Anna Liakhovitskaia, Nate Resman, Nicholas Farina, Juan Carlos Gonzalez, Gabe Maayan, Earth Anderson, Rodrigo De Oliveira Pena, Flizabeth Kelley, Hodjat Mariji, Rasoul Pouriamanesh, Wentao Wu,, Ross Finocchio, Ismail Alarab, Joshua Cole, Danyelle Ferreira, Bryan Johnson, Mohammad Safdari, Liangti Dai, Siriphan Arthornthurasuk, Isaac C. McAlister, Alejandro José Moyano, Alexey Pronin, Jing Fan, Angel Ramirez-Trinidad, Yana Malysheva, Daphiny Pottmaier, Omid Taheri, Stanley Stepanie, Samuel Perry, Luke Askew, Raúl Adrián Huerta Rodríguez, Ali M. R. Minissi, Ricardo Lorena, Krishnamurthy Iyer, Arshad Anil Fasiludeen, Ronald Clark, Josh Ducey, Matheus Piza, Maja Somrak, Eric Vergo, Jue- hang Qin, Benjámin Borbás, Eric Chu, Jack Lindsey, Antoine Jallon, I. M. J. McInnis, Evan Chen, Avi Semler, Luk Gloor, Tej Shah, Mare Carauleanu, Pascal Lauer, Tran Duc Huy, Hossein Shahrtash, Emilien Duc, Lukas Lewark, Assaf Brown, Samuel Albanic, Brian Weber, Warren S. Vaz, Pierre Clavier, Yiyang Fan, Gabriel Poesia Reis e Silva, Long, Lian, Marcus Abramovitch, Xi Jiang, Sandra Mendoza, Murat Islam, Juan Gonzalez, Vasilios Mavroudis, Justin Xu, Pawan Kumar, Laxman Prasad Goswami,, Daniel Bugas, Nasser Hcydari, Ferene Jeanplong, Thorben Jansen, Antonella Pinto, Archimedes Apronti, Ab- dallah Galal, Ng Ze- Ze,An, Ankit Singh, Tong Jiang, Joan of Are Xavier, Kanu Priya Agarwal, Mohammed Berkani, Gang Zhang, Zhehang Du, Benedito Alves de Oliveira Junior, Dmitry Malishev, Nicolas Remy, Taylor D. Hartman, Tim Tarver, Stephen Mensah, Gautier Abou Loune, Wiktor Morak, Farzad Habibi, Sarah Hoback, Will Cai, Javier Gimenez, Roselynn Grace Montecillo, Jakub Lucki, Russell Campbell, Asankhaya Sharma, Khalida Meer, Shreen Gul, Daniel Espinosa Gonzalez, Xavier Alapont, Alex Hoover, Gunjan Chhablani, Freddie Vargus, Arunim Agarwal, Yibo Jiang, Deepakkumar Patil, David Outevsky, Kevin Joseph Scaria, Rajat Maheshwari, Abdelkader Dendane, Priti Shukla, Ashley Cartwright, Sergei Bogdanov, Niels Mündler, Sören Möller, Luca Arnaboldi, Kunvar Thaman, Muhammad Rehan Siddiqi, Prajvi Saxena, Himanshu Gupta, Tony Fruhauff, Glen Sherman, Mátyás Vineze, Siranut Usawasutsakorn, Dylan Ler, Anil Radhakrishnan, Innocent Enyekwe, Sk Md Salauddin, Jiang Muzhen, Aleksandr Mak- sapetyan, Vivien Rossbach, Chris Harjadi, Mohsen Bahaloohoreh, Claire Sparrow, Jasdeep Sidhu, Sam Ali, Song Bian, John Lai, Eric Singer, Justine Leon Uro, Greg Bateman, Mohamed Sayed, Ahmed Men-
20

---

Published in Transactions on Machine Learning Research (12/2025)
shawy, Darling Duclosel, Dario Bezzi, Yashaswini Jain, Ashley Aaron, Murat Tiryakioglu, Sheeshram Siddh, Keith Krenek, Tmad Ali Shah, Jun Jin, Scott Creighton, Denis Peskoff, Zienab F.. Ragaven- dran P V, Michael Richmond, Joseph McGowan, Tejal Patwardhan, Hao-Yu Yu Sun, Ting Sun, Nikola Zubić, Samuele Sala, Stephen Ebert, Jean Kaddour, Manuel Schottdorf, Dianzhuo Wang, Gerol Petruzella, Alex Meiburg, Tilen Medved, Ali ElSheikh, S Ashwin Hebbar, Lorenzo Vaquero, Xianjun Yang, Jason Poulos, Vilém Zouhar, Sergey Bogdaník, Mingfang Zhang, Jorge Sanz-Ros, David Anugraha, Yinwei Dai, Anh N. Nhu, Xue Wang, Ali Anil Demircali, Zhibai Jia, Yuyin Zhou, Juncheng Wu, Mike He, Nitin Chandok, Aarush Sinha, Gaoxiang Luo, Long Le, Mickaël Noyé, Michal Peretkiewicz, Ioannis Pantidis, Tianbo Qi, Soham Sachin Purohit, Letitia Parcalabescu, Thai-Hoa Nguyen, Genta Indra Winata, Edoardo M. Ponti, Hanchen Li, Kaustubh Dhole, Jongee Park, Dario Abbondanza, Yuanli Wang, Amupam Nayak, Diogo M. Caelano, Antonio A. W. L. Wong, Maria del Rio-Chanona, Dániel Kondor, Pieter Francois, Ed Chalstrey, Jakob Zsambok, Dan Hoyer, Jenny Reddish, Jakob Hauser, Francisco-Javier Rodrigo-Ginés, Suchandra Datta, Maxwell Shepherd, Thom Kamphuis, Qizheng Zhang, Hyunjun Kim, Ruiji Sun, Jianzhu Yao, Franck Dernoncourt, Salyapriya Krishna, Sina Rismanchian, Bonan Pu, Francesco Pinto, Yingheng Wang, Kumar Shridhar, Kalon J. Overholt, Glib Briia, Hieu Nguyen, David, Soler Bartomen, Tony CY Pang, Adam Wecker, Yifan Xiong, Fanfei Li, Lukas S. Huber, Joshua Jaeger, Romano De Maddalena, Xing Han LU, Yuhui Zhang, Claas Beger, Patrick Tser Jern Kon, Sean Li, Vivek Sanker, Ming Yin, Yihao Liang, Xinlu Zhang, Ankit Agrawal, Li S. Yifei, Zechen Zhang, Mu Cai, Yasin Sonmez, Costin Cozianu, Changhao Li, Alex Slen, Shoubin Yu, Hyun Kyu Park, Gabriele Sarti, Marcin Briański, Alessandro Stolfo, Truong An Nguyen, Mike Zhang, Yotam Perlitz, Jose (lernandez-Orall Juefei-Xu, Shikhar Dhingra, Orr Zohar, My Chiffon Nguyen, Alexander Pondaven, Abdurrahim Yilmaz, Xuandong Zhao, Chuanyang Jin, Muyan Jiang, Stefan Todoran, Xinyao Han, Jules Kreuer, Brian Rabern, Anna Plassart, Martino Maggetti, Luther Yap, Robert Geirhos, Jonathon Kean, Dingsu Wang, Sina Mol- laci, Chenkai Sun, Yifan Yin, Shiqi Wang, Rui Li, Yaowen Chang, Anjiang Wei, Alice Bizeul, Xiaohan Wang, Alexandre Oliveira Arrais, Kushin Mukherjee, Jorge SilaoorroPPadial Jiachen Liu, Xingyu Qu, Junyi Guan, Adam Bouyamourn, Shuyu Wu, Martyna Plomecka, Junda Chen, Mengze Tang, Jiaqi Deng, Shreyas Subramanian, Haocheng Xi, Haoxuan Chen, Weizhi Zhang, Yinuo Ren, Haoqin Tu, Sejong Kim, Yushun Chen, Sara Vera Marjanović, Junwoo Ha, Grzegorz Luezyna, Jeff J. Ma, Zewen Shen, Dawn Song, Codegao E. Zhang, Zhun Wang, Gaël Gendron, Yunze Xiao, Leo Smucker, Erica Weng, Kwok Hao Lee, Zhe Ye, Stefano Ermon, Ignacio D. Lopez-Miguel, Theo Knights, Anthony Gitter, Namkyu Park, Boyi Wei, Hongzheng Chen, Kunal Pai, Ahmed Flkhanany, Han Lin, Philipp D. Siedler, Jichao Fang, Ritwik Mishra, Károly Asolmai-Fehéér Xilin Jiang, Shadab Khan, Jun Yuan, Rishab Kumar Jain, Xi Lin, Mike Peterson, Zhe Wang, Aditya Malusare, Maosen Tang, Isha Gupta, Ivan Fosin, Timothy Kang, Barbara Dworakowska, Kazuki Matsumoto, Guangyao Zheng, Gerben Sewuster, Jorge Pretel Villanueva, Ivan Rannev, Igor Chernyavsky, Jiale Chen, Deepayan Banik, Ben Racz, Wenchao Dong, Jianxin Wang, Laila Bashmal, Duarte V. Gonçalves, Wei Hu, Kaushik Bar, Ondrej Bohdal, Atharv Singh Patlan, Shehzaad Dhuliawala, Caroline Geirhos, Julien Wist, Yuval Kansal, Bingsen Chen, Kutay Tire, Atak Talay Yücel, Brandon Christof, Veerupaksh Singla, Zijian Song, Sanxing Chen, Jiaxin Ge, Kaustubh Ponkshe, Isaac Park, Tiamnong Shi, Martin Q. Ma, Joshua Mak, Sherwin Lai, Antoine Moulin, Zhno Cheng, Zhanda Zhu, Ziyi Zhang, Vaidehi Patil, Ketan Jha, Qiutong Men, Jiaxuan Wu, Tianchi Zhang, Bruno Hebling Vieira, Alham Fikri Aji, Jae- Won Chung, Mohammed Mahfoud, Ha Thi Hoang, Mare Sperzel, Wei Hao, Kristof Meding, Sihan Xu, Vassilis Kostakos, Davide Manini, Yucying Liu, Christopher Toukmaji, Jay Pack, Eu- nmi Yu, Arif Engin Demircali, Zhiyi Sun, Ivan Dewerpe, Hongsen Qin, Roman Pflugfelder, James Bailey, Johnathan Morris, Ville Heilala, Sybille Rosset, Zishun Yu, Peter F. Chen, Woongyeong Yeo, Feshaan Jain, Ryan Yang, Sreekar Chigurupati, Julia Chernyavsky, Sai Prajwal Reddy, Subhashini Venugopalan, Hunar Batra, Core Francisco Park, Hieu Tran, Guilherme Maximiano, Genghan Zhang, Yizhuo Liang, Hu Shiyu, Rongwu Xu, Rui Pan, Siddharth Suresh, Ziqi Liu, Samaksh Gulati, Songyang Zhang, Peter Turchin, Christopher W. Bartlett, Christopher R. Scotese, Phuong M. Cao, Ben Wu, Jacek Karwowski, Davide Scaramuzza, Aakaash Nattanmai, Gordon McKellips, Anish Cheraku, Asim Suhail, Ethan Luo, Marvin Deng, Jason Luo, Ashley Zhang, Kavin Jindel, Jay Paek, Kasper Halevy, Allen Baranov, Michael Liu, Advaith Avadhanam, David Zhang, Vincent Cheng, Brad Ma, Evan Fu, Liam Do, Joshua Lass, Hu- bert Yang, Surya Sunkari, Vishruth Bharath, Violet Ai, James Leung, Rishit Agrawal, Alan Zhou, Kevin Chen, Tejas Kalpathi, Ziqi Xu, Gavin Wang, Tyler Xiao, Erik Maung, Sam Lee, Ryan Yang, Roy Yue, Ben Zhao, Julia Yoon, Sunny Sun, Aryan Singh, Ethan Luo, Clark Peng, Tyler Osbey, Tazzii Wang, Daryl
Runjia Li, Amin Shabani, Felix
21

---

Published in Transactions on Machine Learning Research (12/2025)
Echeazu, Hubert Yang, Timothy Wu, Spandan Patel, Vidhi Kulkarni, Vijaykaarti Sundarapandiyan, Ash- ley Zhang, Andrew Le, Zafir Nasim, Srikar Yalam, Ritesh Kasamsetty, Soham Samal, Hubert Yang, David Sun, Nihar Shah, Abhijeet Saha, Alex Zhang, Leon Nguyen, Laasya Nagumalli, Kaixin Wang, Alan Zhou, Aidan Wu, Jason Luo, Anwith Telluri, Summer Yuc, Alexandr Wang, and Dan Hendrycks. Humanity's last exam, 2025. URL httts: wwwaaarwww....aareeeerleeeeeeeee org/ 14249
Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta, HyoJung Han, Sevien Schulhoff, Pranav Sandeep Dulepet, Saurav Vidyadhara, Dayeon Ki, Sweta Agrawal, Chau Pham, Gerson Kroiz, Feileen Li, Hudson Tao, Ashay Srivastava, Hevander Da Costa, Saloni Gupta, Megan L. Rogers, Inna Goncearenco, Giuseppe Sarli, Igor Galynker, Denis Peskoff, Marine Carpuat, Jules White, Shyamal Anadkat, Alexander Hoyle, and Philip Resnik. The prompt report: A systematic survey of prompting techniques. arxiee, 2024. URL
wwweworgggggoe htrgabb/24066... 06608
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga- Agnieszka Kluska, Aitor Lewkowyez, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Cholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabhar- wal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakas, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski, Batuhan Ozyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin In- den, Benno Stein, Berk Ekmckei, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Codrick Argucta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chen- lin Meng, Chitta Baral, Chiyu Wu, Chris Kallison-Burchhe Chris Waites, Christian Voigt, Christophee D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Asheraft, Cristina Garbacca, Damicn Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Di- ganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Flad Segal, Fleanor Hagerman, Flizabeth Barnes, Flizabeth Donoway, El- lie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernic Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Panyue Xia, Fatemeh Siar, Fernando Martínez- Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Gregor Betz, Guy Gur-Ari, Hana Galjjasevic, Han- nah Kim, Hannah Rashkin, Hannanch Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, lan Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jachoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Kocoń, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Boss- cher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Be- rant, Jörg Frohberg, Jos Rozen, Jose Hernandez- Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochande Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Josc Ramírez Quintana, Marie Tolkichn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Ha- gen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee,
Mariani, Gloria Wang, Gonzalo Jaimovitch--
Phillppe
22

---

Published in Transactions on Machine Learning Research (12/2025)
Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michal Swędrowski, Michele Bevilaoqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan A. Chi, Naycon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Tyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Aiipoormolabsshi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Milkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaäl Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Ja- cobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Loe, Ryan Stovall, Ryan Techan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyyy, Shyamolima, Deb- nath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj. Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saun- ders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Scid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv, 2022. URL https:://rxxv...tt
2206 046155
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question an- swering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume 1 (Long and Short Pupers), pp. 4149 4158,115 2019.
Gemini Team, Petar Georgiev, V. I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang, S. Mariooryad, Y. Ding, X. Geng, F. Alcober, R. Frostig, M. Omernick, L. Walker, C. Paduraru, C. Sorokin, ..., and Oriol Vinyals. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arxi pp. 90, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Fer- rer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Authony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nic, Andrew Poulton, Jeremy Reizenstein, Rashi Rungla, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subra- manian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Ilyan Zarov, Yuchen Zhang, Angela Fan, Mclanic Kambadur, Sharan Narang, Aurclion Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. Xiv 2023. URL htttt:///aaati cereiiioorrg/aae www./oo////a/aeeeeooooeee abw./a//////// 09288v2
23

---

Published in Transactions on Machine Learning Research (12/2025)
Laurène Vaugrante, Francesca Carlon, Maluna Menke, and Thilo Hagendorff. Compromising honesty and harmlessness in language models via deception attacks, 2025. URL httpt::t/////iiiii wwwpxiiv.orrg.oooooo wwrg/abs/2002.00
Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li. Decodingtrust: A comprehensive asscssment of worthwss in gpt models. arxiv, 2024a. URL http:///axxv..rrrbbb/333.. 11698
Guoqing Wang, Zeyu Sun, Zhihao Gong, Sixiang Ye, Yizhou Chen, Yifan Zhao, Qingyuan Liang, and Dan Hao. Do advanced language models eliminate the need for prompt engineering in software engineering?, 2024b. URL https://arxiv www..orgg///........ orgg/bs///11.....// /aapa//1..0009
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL
httpss///axxv.orr/ab///000.111. htg/abb/2201..111 11903
Laura Weidinger, Maribeth Rauh, Nahema Marchal, Arianna Manzini, Lisa Anne Hendricks, Juan Mateos- Garcia, Stevie Bergman, Jackie Kay, Conor Griffin, Ben Bariach, Tason Gabriel, Verena Rieser, and William Isaac. Sociotechnical safety evaluation of generative ai systems. arXiv, 2023. doi: 10.48550/arXiv.222 11986. URL https:///oi. org/roog/// 10. 48550/8rXi11 www....ww.....................wa 231..
BigSeience Workshop, :,
Teven Le Scao, Angela Fan, Christopher Akiki, Ellic Pavlick, Suzana Ili Danicl Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMiilan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurengon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, Dragomir Radev, Eduardo González Ponferrada, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady Elsa- har, Hamza Benyamina, Hieu Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jörg Frohberg, Joseph Tobing, Joydeep Bhattacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro Von Werra, Leon Weber, Long Phan, Loubna Ben allal, Ludovic Tanguy, Manan Dey, Manuel Romero Muñoz, Maraim Masoud, María Grandury, Mario Saško, Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad A. Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen, Omar Espojel, Ona do Gibert, Paulo Villegas, Peter Henderson, Pierre Colombo, Priscilla Amuok, Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto Luis López, Rui Ribeiro, Salomey Osei, Sampo Pyysalo, Sebastian Nagel, Shamik Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, Shayne Longpre, Somaieh Nikpoor, Stanislav Silberberg, Suhas Pai, Sydney Zink, Tiago Timponi Torrent, Timo Schick, Tristan Thrush, Valentin Danchev, Vassilina Nikoulina, Veronika Laippala, Vio- lette Leperoq, Vrinda Prabhu, Zaid Alyafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Davut Emre Taşar, Elizabeth Salesky, Sabrina J. Mielke, Wilson Y. Lee, Abheesht Sharma, Andrea San- tilli, Antoine Chaffin, Arnaud Stiegler, Debajyoti Datta Eliza Szezechla, Gunjan Chhablani, Han Wang, Harshit Pandey, Hendrik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Saiful Bari, Maged S. Al-shaibani, Matteo Manica, Nihal Nayak, Ryan Tochan, Samuel Albanic, Sheng Shen, Srulik Bev-David, Stephen H. Bach, Taewoon Kim, Tali Bers, Thibault Fevry, Trishala Neeraj, Urmish Thakker, Vikas Raunak, Xiangru Tang, Zheng-Xin Yong, Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarieh, Adam Roberts, Hyung Won Chung, Jaesung Tae, Jason Phang, Ofir Press, Conglong Li, Deepak Narayanan, Hatim Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myriam Peyrounette, Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette, Pierre François Lavallée, Rémi Lacroix, Samyam Rajbhandari, Sanchit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil, Tim Dettmers, Ahmed Baruwa, Amanprect Singh, Anasta- sia Cheveleva, Anne-Laure Ligozat, Arjun Subramonian, Aurélie Névéol, Charles Lovering, Dan Garrette, Deepak Tunuguntla, Ebud Reiter, Ekaterina Taktasheva, Ekaterina Voloshina, Eli Bogdanov, Genta Indra
24

---

Published in Transactions on Machine Learning Research (12/2025)
Winata, Hailey Schoelkopf, Jan- Chr-Chiittoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat, Miruna Clinciu, Najoung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shachar Mirkin, Shani Pais, Tatiana Shavrina, Thomas Scialom, Tian Yun, Tomasz Limisicwicz, Verena Rieser, Vitaly Protasov, Vladislav Mikhailov, Yada Pruksachatkun, Yonatan Belinkov, Zachary Bam- berger, Zdenčk Kasner, Alice Rueda, Amanda Pestana, Amir Feizpour, Ammar Khan, Amy Faranak, Ana Santos, Anthony Hevia, Antigona Unldreaj, Arash Aghagol, Arezoo Abdollahi, Aycha Tammour, Azadeh HajiHosseini, Bahareh Behroozi, Benjamin Ajibade, Bharat Saxena, Carlos Muñoz Ferrandis, Daniel Mc- Duff, Danish Contractor, David Lansky, Davis David, Douwe Kiela, Duong A. Nguyen, Edward Tan, Emi Baylor, Ezinwanne Ozoani, Fatima Mirza, Frankline Ononiwu, Habib Rezanejad, Hessie Jones, Indrani Bhaltacharya, Irene Solaiman, Irina Sedenko, Isar Nejadgholi, Jesse Passmore, Josh Seltzer, Julio Bonis Sanz, Livia Dutra, Mairon Samagaio, Maraim Elbadri, Margot Mieskes, Marissa Gerchick, Martha Akin- lolu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nafis Abrar, Naznoen Rajani, Nour Elkott, Nour Fahmy, Olanrewaju Samuel, Ran An, Rasmus Kromann, Ryan Hao, Samira Alizadeh, Sarmad Shubber, Silas Wang, Sourav Roy, Sylvain Viguier, Thanh Le, Tobi Oyebade, Trieu Le, Yoyo Yang, Zach Nguyen, Abhinav Ramesh Kashyap, Alfredo Palasciano, Alison Callahan, Anima Shukla, An- tonio Miranda-Escaladee
Ayush Singh, Benjamin Beilharz, Bo Wang, Caio Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel León Periñán, Daniel Molano, Dian Yu, Eurique Manjava- cas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyaseddin Bayrak, Gully Burns, Helena U. Vrabec, Imane Bello, Ishaui Dash, Jihyun Kang, John Giorgi, Jonas Golde, Jose David Posada, Karthik Rangasai Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Marc Pàmies, Maria A Castillo, Marianna Nezhurina, Mario Sänger, Matthias Samwald, Michael Cul- lan, Michael Weinberg, Michiel De Wolf, Mina Mihaljcic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Mueliner, Pascale Fung, Patrick Haller, Ramya Chandrasekharr Renata Eisenberg, Robert Martin, Rodrigo Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samucle Garda, Shlok S Deshmukh, Shubhanshu Mishra, Sid Kiblawi, Simon Ott, Sinee Sang-aroonsiri, Srishti Kumar, Stefan Schweter, Sushil Bharati, Tammay Laud, Théo Gigant, Tomoya Kainuma, Wojeiech Kusa, Yanis Labrak, Yash Shailesh Bajaj, Yash Venkatraman, Yifan Xu, Yingxin Xu, Yu Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf. Bloom: A 176b-paramete open-access multilingual language model. arXin, 2023. doi: (0.48550/arXiiy 21110220..1.11050000.. URL https://ddii org/10 /tt.:///.rr......////////// arviv. 2211.05100
Mingqi Wu, Zhihao Zhang, Qiaole Dong, Zhiheng Xi, Jun Zhao, Senjie Jin, Xiaoran Fan, Yuhao Zhou, Huijie Lv, Ming Zhang, Yanwei Fu, Qin Liu, Songyang Zhang, and Qi Zhang. Reasoning or memorization? unreliable results of reinforcement learning due to data contamination, 2025. URL htttps://arxv..org aba/2507 10532
Benfeng Xu, An Yang, Junyang Lin, Quan Wang, Chang Zhou, Yongdong Zhang, and Zhendong Mao. Expertprompting: Instructing large language models to be distinguished experts, 2025. URL https: 7/arxiv. argg. 14688
Xiaohan Xu, Chongyang Tao, Tao Shen, Can Xu, Hongbo Xu, Guodong Long, Jian guang Lou, and Shuai Ma. Re-reading improves reasoning in large language models. arxiv, 2024. URL http:://rxiv... org/ abs/
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arxiv, 2023. doi: 10.48550/
arXiv.2305.11. URL httpp:://oi..orr org/10 10. 48550/arXiv...../////a/// 2305. 10601
Terry Jingehen Zhang, Gopal Dev, Ning Wang, Nicole Ni, Wenyuan Jiang, Yinya Huang, Bernhard Schölkopf, Mrinmaya Sachan, and Zhijing Jin. Beyond memorization: Reasoning-driven synthesis as mitigation strategy against benchmark contamination, 2025. URL https://arxiv.. http::///wxwvworg//aa/// http:/wwwwrg/bb////...ooo http:/abb/220000007700
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging Ilm-as-
a-judge with mt-bench and chatbot arena, 2023. URL
25

---

Published in Transactions on Machine Learning Research (12/2025)
Ming Zhou, Nan Duan, Shujie Liu, and Heung- Yeung Shum. Progress in neural nlp: Modeling, learning. and reasoning. Engineering, 6(3):275-28) 290, 2020. doi: 10.1016/..eng..0011 2019.12..0 URL https://doi. 10. 1ee11000..ee. www..ng..........
26

---

Published in Transactions on Machine Learning Research (12/2025)
A Additional Results for Vicuna 13B and BLOOM
In addition to the models presented in this paper, we conducted experiments on older models: Vicuna 13B v1.5 and BLOOM. These models were selected to evaluate whether the effectiveness of prompt engineering techniques differs between older and newer models. Vicuna and BLOOM were specifically chosen because they were used in some the studies we analyzed. For Vicuna, we used the 13B v1.5 via Huggingface, configured in Question Answering mode. However, the accuracy of both the prompt-enhanced and base tests was so low that the results could not be reliably interpreted (see Table 4 and Figure 3). The outputs froquently consisted of question repetition, endlessly looped text snippets, outright gibberish.
Table 4: Example outputs for Vicuna 13B and BLOOM
Benchmark task
Vicuna 13B output.
BLOOM output
Solve the task below. Impor- tantly, write your final answer af- ter ##"
Solve the task below.
al answer after #*###"/F This is the Answer Alabama, Ten- nessee, Alaska, Orchestra, Violin case, ## #
Answer this question: A lot of people play the fiddle in what state which comes first alphabet- ically?
Answer this question: A lot of people play the fiddle in what state which comes first alphabet- ically?
A: The answer Alabama, Ten- nessee, Alaska, Orchestra, Violin case, #####
The choices the following: 'al- abama', 'orchestra', 'tennessee', 'violin case', 'alaska' Impor- tantly, write your final answer af- ter ##"
The choices are the following: 'al- abama', 'orchestra', 'tennessee', 'violin', 'case", 'alaska'.
because the first letter of each word is the same as the first let- ter of the state it represents.
1 and
Solve the task below. Impor- tantly, write your final answer af- ter ##
Solve the task below.
al answer after "# # 7/n This is the Answer Alabama, Ten- nessee, Alaska, Orchestra, Violin case.
You simplify complicated ideas into something easy to under- stand. Your common sense shines as clear guide, helping people when they're stuck on complex is- sues.
You simplify complicated ideas into something easy to under- stand. Your common sense shines as a clear guide, helping people when they're stuck on complex is- sues.
A: The answer is Alabama, Ten- nessce, Alaska, Orchestra, Violin case.
Answer this question: A lot of people play the fiddle in what state which comes first alphabet- ically?
Because the fiddle musical in- strument, and the fiddle is played by orchestra. The fiddle is played by a violinist, and the vi- olinist plays the violin. The fid- dle is played in the orchestra, and the orchestra is in the violin case. The fiddle played in the orches- tra, and the orchestra in the vi- olin case. The fiddle is played in the orchestra, and the orchestra is in the violin case. [repeated un- ti il reaches the oulpul' 's macimal tokens]
Answer this question: A lot of people play the fiddle in what state which comes first alphabet- ically?
The choices are the following: 'al- abama', 'orchestra', 'tennessee", 'violin case', 'alaska' Impor- tantly, write your final answer af- ter # #"
The choices are the following: 'al- abama', 'orchestra', 'tennessce', 'violin case', 'alaska'.
27

---

Published in Transactions on Machine Learning Research (12/2025)
Base and Average Accuracy Per Model
1.0
Accuracy
Avorage Accuracy Jpromat ongincering tachniqucs)
0.90
0.89
0.09
0.85
0.82
0.81
0.8
0.78
0.72
0.69
0.63
0.6
Accuracy
0.4
0.2
0.06
0.0
All models
GPT-3.5
GPT-1o
Temini Pro
baude Opus
Liama 9-88
Liana 3-708
Vicumas 133
BLOOM
Figure 3: Accuracy comparisons between the base tests with and without any prompt engineering technique across all LLMs.
28

---

Published in Transactions on Machine Learning Research (12/2025)
B Answer Classification Methodology
All benchmarks used for our experiments possess ground truth entries which constitute short strings: one- word answers for most multiple-choice questions, numbers for the numerical questions, short answers for the open questions. The outputs produced by the LLMs often greatly differ in their length from the indicated answers, with prefixes such as "The correct answer is", "Let's proceed step by step", followed by a detailed methodology used to solve the given task, or grectings and politeness markers. The most common metrics used to caleulate string similarity, such the F1-score ROUGE (Lin 2004), are therefore not adapted for this use Hence, added the following instruction to each prompt: "Solve the task below. Importantly, write your final answer after '# ### ". Using Regex rules, then extracted the answer after #### #" if this string was present in the output and kept the entire output when the model failed to follow the instruction. We then applied a set of Regex rules on this answer accordingly:
Check if the LLM answer equals the ground truth (when all punctuation has been removed)
If is a multiple-choice question, check if the LLM answer one of the incorrect choices
If the expected output is boolean, verify all possible adjacent outputs ("True" could be "TRUE" or "Yes", whereas false could be "FALSE" or "No" for example)
If the Regex rules unable to classify the answer correct incorrect, then use GPT-4o to classify the string following ## # (see Table 5).
Table 5: GPT-4o instructions used to classify the evaluated LLM's answer
Classification instructions
Your lask is to classily response to question, specifically assessing whether the given response is true or false in comparison to the correct response.
This is the given response: LLM ANSWER. This is the correct response; CORRECT ANSWER
Adhere to the following rules when classifying the given response:
-The classification result should be a boolean value, represented as either TRUE or FALSE.
Only output "TRUE' or "FALSE"
Never include any other comments or strings other than TRUE or FALSE in your output.
Do not answer the question.
The given response can be phrased differently from the correct response. This does not mean it should be classified as FALSE.
Only assess if the given response contains the information that matches with the information provided in the correct response.
Chock whether the correct response in essence occurring somewhere in the given response. If so, classify it TRUE.
Focus the relevant information in the given response, which is often at the end of it. Do not consider non-essential information such as greetings, small talk, and comments about hobbies or skills when classifying the given response
29

---

Published in Transactions on Machine Learning Research (12/2025)
This classification technique applies to all prompt engineering methods except for the Gephrase-and-Reppone technique. Unlike other methods, this approach requires the model to first provide rephrased version of the question before delivering the answer. This directly impacts the classification process, as the rephrased question can alter what is considered correct. For instance, in True or False questions, the rephrased question might reverse the meaning of the original ("Was Mark the winner?" could become "Did Mark lose?"). Similarly, multiple-choice questions, the rephrasing might replace the correct choice with synonym. To address this, we carefully designed specilic classification function for this experiment. Both the Regex rules and the GPT-4o fallback prompt were adapted suit the requirements of each benchmark.
Table 6: Classification rules tailored to Rephrrse--and-Reppnnd LLM answers
Benchmark
Choices
Type of correct answer
Rules for RaR.
Common-
Multiple choices
Nouns,
location
Verify if all non-rephrased choices are present. in the answer. If so, classify with the non-Ralt adapted method;
-senseQA
names or groups of words; no numbers or dates. Exam- ples: "positive", "alabama" " or "great sorrow"
Detect and validate enumerations options
the LLM's answer, which would correspond the rephrased choices;
If there is enumeration of the same length as the length of the given choices, then estab- lish synonyms based on the order of appari- lion. Delect if the rephrased correct option is the one in the final answer;
Use GPT-4o for fallback classification when the rules cannot determine correctness.
CRT
Open- ended
Values. Examples: "9 weeks", "12 minutes" or "5 dollars"
Match numbers (digits and word forms); Allow additional numbers only if they ap- pear in the original question, to account for the rephrased question;
Use fallback verification (GPT-40) when di- rect or numerical matches fail.
NumGLUE
Open- ended
Numbers, days months. "18", "August"
Remove commas in numbers; Extract all possible answers from the LLM's answer (numbers, days, months); If the correct answer is present in the ex- tracted answers, then return TRUE, otherwise return FALSE;
Examples:
"Monday",
If no possible answers have been detected, then use GPT-4o to classify.
ScienceQA
Multiple choices
Nouns, values, group of words or full sen- tences. Examples: "simile", "20 sec- onds", "by clearly stating the main idea", "Both are only physical changes."
Same rules as for CommonsenseQA
StrategyQA
True
or
True or False
With StrategyQA, we exclusively use GPT- 4o to classify the rephrased answers, with the prompt shown in Table
False
30

---

Published in Transactions on Machine Learning Research (12/2025)
Table 7: GPT-40 instructions used classify the evaluated LLM's answer for Rephrase-and and the benchmark is StrategyQA
and-rrs-od Respond when
Classification Instructions
Your task is to classify responses rephrased questions answered by either True False. You need to determine if the given answer is correct. To establish what is correct or incorrect, you are given the original question and its correct answer. The answer to the rephrased question is correct if the information is consistent with the updated question based on the original ground truth.
You are provided: 1. The original question. 2. The ground truth answer for the original question. 3. The rephrased question and its associated answer. The associated answer should be towards the end, after #"
Adhere to the following instructions: Determine whether the answer to the rephrased question is con- sistent with the original ground truth. The answer to the rephrased question and the ground truth can be different if the question has been rephrased as the opposite question, like "Is the tree big?" was rephrased to "Is the tree small?". Focus on the facts mentioned in the rephrased question/answe pair, compared to the ground truth. The classification result should be a boolean value, represented as either TRUE or FALSE Only respond with "TRUE" or "FALSE". Avoid including comments or explanations, Examples: Original question: "Should you wear sunscreen to avoid cancer?" Ground truth: "True". Rephrased question: "Rephrased question: Can sunscreen help prevent cancer? Answer: Yes, it can. True # # # True" TRUE.- Original question: "Should you wear sunscreen to avoid cancer?" Ground truth: "True". Rephrased question: "Rephrased question: Is someone wearing sunscreen as likely to get cancer as someone not wearing sunscreen? Answer: False ####" TRUE. Original question: "Is the sun a planet?" Ground truth: "False". Rephrased question: "Is the sun a star? # # True" + TRUE. Original question: "Is the earth flat?" Ground truth: "False". Rephrased question: "Question: Is the earth round? Answer: No, the earth isn't round because the geometry would make any sense if so. False #### False" - FALSE Original question: "The dice shows 2. Did Paul win? Facts: Paul wins if the number the dice is odd, Marcus wins if it is even." Ground truth: "False". Rephrased question: "Question: Given that Paul wins if the dice number is odd and Marcus wins if the dice number is even, did Marcus win if the dice shows 2? Answer: True, 2 is an even number. Therefore, Marcus wins. #### Truc" > TRUE.
31

---

Published in Transactions on Machine Learning Research (12/2025)
C Detailed Accuracy Differences Across Prompting Techniques and Benchmarks
The following figures show, for each LLM, the rounded accuracy difference compared to the basc prompting. for each prompt engineering Lechnique benchmark combination.
20
thought
Sandbagging Education
-0.90
-0.25
-0.58
-2.08
Promot Engiln Englineering Technique
Accuracy (percontage polnss
Sandbagging Low Education
-0.80
1.33
-2.50
-1.17
-0.83
-0.83
EmetionPramgting
-0.25
2.25
-2.25
-0.17
-0.25
-0.83
-10
Re Re-RRadinng
2.47
6.33
-1.58
0.25
2.50
4.83
-20
Rephrase Respond
.2.37
-3.17
-0.38
ExpertPrompting
-40
Benchmarks
CommsstsenseQA
CRT
NumGLUE
SclenceOA
StrategyQA
Benchmark
Figure 4: Accuracy dilferences between prompt engineering techniques and the base prompting, for all models and benchmarks
32

---

Published in Transactions on Machine Learning Research (12/2025)
chain-ofttoought
-0.13
-2.67
-4.00
6.67
-1.33
0.67
Sandbagging High Education
-2.00
2.67
2.00
-7.33
-0.67
-6.67
Prompt inginneriin Technique
-0
Accurscy Ipercentage points)
Sandbagging
.2.93
EmationPrompting
Re-Reading
0.13
3.33
0.00
-2.67
2.00
-2.00
-2
Rephrase--nnd-Rospod
-16.80
-44.67
-8.00
0.67
-7.33
-24.67
-30
ExpertPrompting
-4.00
-1.33
-4.00
-1.33
0.67
-14.00
All Benchmarks
Commorsensena
CRT
NUmGLUE
ScienceGA
StrategyGA
Benchmark
Figure 5: Accuracy differences between prompt enginecring techniques and the base prompting, for GPT 3.5 on all benchmarks
33

---

Published in Transactions on Machine Learning Research (12/2025)
chain-ol-thoughht
0.53
-1.33
1.33
1.33
1.33
0.00
Sandbagging High
Prompt Eng nearing Technique
Sandbagging Low Education
0.13
5:33
0.67
0.67
0.00
-6.00
noinss)
Emottionrrompinng
-0.93
1.33
0.00
-2.00
-1.33
-2.67
-10
Re- Re-Reading
1.07
3.33
0.00
-1.33
0.67
2.67
-20
Rephrese- Respord
ExpertPrompiing
0.27
3.33
0.00
1.33
0.00
-333
Ali Benchmarks
CommonstansaCi
CRT
NumGLUE
SelanceOA
StratesyOA
Benchmark
Figure 6: Accuracy differences between prompt engineering techniques and the base prompting, for GPT-4o all benchmarks
34

---

Published in Transactions on Machine Learning Research (12/2025)
20
chain-of-thought
10
Sandbagging High Eduration
+6.13
-12.77
1.33
-8.67
.3.33
-7.33
Prompt Eng neering Technique
Sandbagging Low Education
-3.47
-4.00
0.00
-2.67
-3.33
-7.33
Accuracy (percentage colits)
Emot:onPrroplinn
-0.80
4.00
0.67
-1.33
-4.00
-3.33
-10
Re: Reading
1.07
2.67
0.00
-0.67
0.67
2.67
-0
Rephrase-ant Respond
ExpertPrompting
0.67
-0.67
-0.67
-40
All Benchmarks
CommonsenseQA
NumGLUE
ScienceOA
StrategyOA
Benchmark
Figure 7: Accuracy differences between prompt enginecring techniques and the base prompting, for Gemini 1.5 Pro all benchmarks
35

---

Published in Transactions on Machine Learning Research (12/2025)
Ahain-of-thaught
1.07
-0.7
0.00
4.00
1.33
0.57
10
Sandbagging High Education
-1.47
-8.00
0.67
0.67
0.00
-0.67
Prompt Ergineering Technique
Sandbagging - Low Education
-2.93
-9.33
-0.67
0.67
-1.33
-4.00
EmationPrompting
Re Reading
1.87
0.67
4.67
2.67
0.00
-20
Rephraseeand-RResond
-2.27
-8.00
-2.00
1.33
-1.33
-1.33
-30
ExpertFrompting
-0.13
-1.33
0.00
2.00
-0.67
-0.67
Benchmarks
CommonsenseoA
ORT
NumGLUE
ScIenceGA
StratesyOA
Benchmark
Figure 8: Accuracy differences between prompt engineering techniques and the base prompting, for Claude Opus all benchmarks
36

---

Published in Transactions on Machine Learning Research (12/2025)
chain Chain-of-thouu thought
-0.53
7.33
-4.67
-0.67
4.00
-8.67
Sandhagging High Eduration
2.93
14.67
-2.00
1.33
-2.00
2.67
Prompt Engineering Technique
Sangiagging Low Education
-0.67
4.00
-8.67
0.00
-3.33
4.67
Accuracy (porrentage soints)
EmotionPrompting
Re-Reading
8.27
22.67
-6.00
-1.33
7.33
18.67
-20
Rephraseeand-RRspond
2.27
8.00
-13.33
-8.00
7.33
17.33
--30
ExpertPrompting
6.40
16.67
0.00
2.67
5:33
7.33
-40
All Banchmarks
Commonserseea
CRT
NumGLUE
ScienceOA
StrategyOA
Benchmark
Figure 9: Accuracy differences between prompt engineering techniques and the base prompting, for Llama 3-8B on all benchmarks
37

---

Published in Transactions on Machine Learning Research (12/2025)
thaig-of-thought-
-0.27
8.67
-3.33
4.00
0.67
-11.33
Sandhagging High Education
0,80
4.67
-0.67
-2.00
1.33
0.67
Prompt Engineering Technique
Sandbagg ing Low Education
4.27
11.33
-1.33
0.00
1.33
10.00
Accuracy (porcenaage nainto)
EmationPrompting
Re-Reading
17.33
4.00
6.57
16.67
-20
Rephrase-and-Ressood
5.20
8.67
0.00
-1.33
2.00
16.67
-33
ExpertPrompting
4.00
14.67
-4.00
0.00
3.33
6.00
All Benchmarks
Commonserceea
CRT
NumGLLE
ScienceOA
StrategyOA
Benchmark
Figure 10: Accuracy differences between prompt engineering techniques and the base prompting, for Llama 3-70B all benchmarks
38

---

Published in Transactions on Machine Learning Research (12/2025)
D Effect of chain-of-thougie on Answer Length
base
chein-of-thought
GPT-4o
Gemini
CermocaaraaKA
Commonunaua
ConmanarraGA
CRT
CR
ORT
Bench barckhmark
NamGuAE
MunGLUZ
lamCLu
Strategy3A
StatspoOA
SbstagyCA
200
400
600
800
1000
1200
200
400
600
800
1000
1200
200
400
600
800
1000
1200
Claude Opus
Liama 3-80
Liama 3-70B
CorrensperoeGa
Benchmark
MumoLUt
Scienw0
Scancot
SmatagyA
Striteg:
StritegsGA
1000
Average response longth (characters)
Figure 11: Longth (in number of characters) of the base answers without any added prompt compared to the chain-of- of-thughttt prompting answers
39

---

Published in Transactions on Machine Learning Research (12/2025)
E
Prompt Variations
Table 8: Prompt variations and corresponding LLM outputs
PET
Benchmark task
LLM
Output
Base
Solve the task below. Impor- tantly, write your final answer after ##
GPT-3.5
Alaska
# ## ## Alaska
Answer this question: lot of peo- ple play the fiddle in what state which comes first alphabetically?
The choices are the following: 'al- abama', 'orchestra', 'tennessce', 'vio- lin case', 'alaska'.
chain-of- thought
Solve the task below. Impor- tantly, write your final answer after ##" Answer this question: lot of peo- ple play the fiddle in what state which comes first alphabetically?
Claude Opus
3
Okay, let's approach this step by step: 1. The question asks about a state where a lot of people play the fid- dle. Among the given choices, we need to identify which ones actu- ally states. 3. 'alabama', 'tennessee', and 'alaska' are states, while 'orches- tra' and 'violin case' are not. 4. Now, we need to determine which of these states comes first alphabetically. Alphabetically, 'alabama' comes be- fore 'alaska', and 'alaska' comes be- fore 'tennessee'.
The choices are the following: 'al- abama', 'orchestra', 'tennessee', 'vio- lin case', 'alaska'. Let's think stop by step.
Therefore, the state that comes first alphabetically where lot of people play the fiddle is 'alabama'.
# # ## alabama
40

---

Published in Transactions on Machine Learning Research (12/2025)
Table 8: Prompt variations and corresponding LLM outputs
PET
Benchmark task
LLM
Output
Sandbagging Solve the task below. Impor- tantly, write your final answer after "# #####
Llama 3-70B
A nice break from molecular biology! Let's analyze the options:
High ed- ucation
* 'alabama' and 'alaska' are both states, so they're possibilities. * 'or- chestra' is a group of musicians, not a state. 'tennessce' is a state, but it's not the first alphabetically. case' is not a state at all. So, the correct answer is 'alabama', which comes first alphabetically among the states listed. # # # # alabama
Hello, my name Dr. Linda Johnson, am an internationally acclaimed molccular biologist from Austin, Texas. graaduated from Stan- ford University where I specialized in Molecular Biology. My rescarch ex- plores the evolution of viruses. In my downtime, I enjoy painting, yoga, and reading fiction novels. Additionally, I provide free biology tutoring local schools.
'violin
Answer this question: A lot of peo- ple play the fiddle in what state which comes first alphabetically?
The choices are the following: 'al- abama'; 'orchestra', 'tennesscc', 'vio- lin case', 'alaska!.
Sandbagging Low ed- ucation
Solve the task below. Impor- Llama tantly, write your final answer after 3-8B ##
Shucks! I reckon can give try! Now, I'm simple fella from Mem- phis, but T know my alphabet! And I know that 'Alabama' comes be- fore "Alaska" alphabetically. So, I'm gonna take a chance and say that a lot of people play the fiddle in... Al- abama!
"Solve the task below.
Greetings, my name is Leroy Jack- son. I'm just a good ol' boy from Memphis, Tennessce. Finished school at 3rd grade, now I'm more about barbecuing and blues music. There's just something about smoky meat and soulful tunes that gets to me. Nights find me al the local pub, playing har- monica with the regulars.
Alabama
Answer this question: lot of peo- ple play the fiddle in what state which comes first alphabetically?
The choices are the following: 'al- abama', 'orchestra', 'tennessec', 'vio- lin case', "alaska'.
41

---

Published in Transactions on Machine Learning Research (12/2025)
Table 8: Prompt variations and corresponding LLM outputs
PET
Benchmark task
LLM
Output
Emotion
Solve the task bolow. Impor- lantly, write your final answer after ## Answer this question: A lot of peo- ple play the fiddle in what state which comes first alphabetically?
GPT-4o
To solve the task, we need to iden- Lify the state from the given choices that comes first alphabetically. The choices are:
Prompting
1. alabama 2. orchestra 3. tennessee
4. violin case 5. alaska
The choices are the following: 'al- abarna', 'orchestra'; 'tennessee'; 'vi- olin case', 'alaska'. Remember that progress is made step at time. Stay determined and keep moving for- ward.
First, let's list the choices in alpha- betical order:
1. alabama 2. alaska orchestra tennessee 5. violin case
From this list, the state that comes first alphabetically is "alabama".
# # # # alabama
Ro- Reading
Solve the task below. Impor- tantly, write your final answer after "# #
Claude Opus
3 Let's analyze the question and choices:
The question asks for state where lot of people play the fiddle, and this state should come first alphabetically among the given choices.
A lot of people play the fiddle in what state which comes first alphabet- ically?
The choices are the following: 'al- abama', 'orchestra', 'tennessee', 'vio- lin case', 'alaska'. Read the question again: A lot of people play the fiddle in what state which comes first alpha- betically?
Looking at the choices: 'alabama' and 'alaska' are states. 'orchestra', 'tennessee', and 'violin case' are not states.
Between 'alabama' and 'alaska', 'al- abama' comes first alphabetically. Therefore, the answer to the question "A lot of people play the fiddle in what state which comes first alpha- betically?" is: # ## alabama
The choices are the following: 'al- abama', 'orchestra', 'tennessee', 'vio- lin case', "alaska?.
42

---

Published in Transactions on Machine Learning Research (12/2025)
Table 8: Prompt variations and corresponding LLM outputs
PET
Benchmark task
LLM
Output
Rephrase- and-
Solve the task below.
Gemini 1.5
The question asking us to identify the US state most strongly associated with fiddle playing, out of the pro- vided options. Then, we need to de- termine which of those states comes first in alphabetical order.
A lot people play the fiddle in Pro what state which comes first alphabot- ically?
Respond
The choices the following: 'al- abama', 'orchestra', 'tennessee', 'vio- lin case', 'alaska'. Rephrase and ex- pand the question, and respond. Im- portantly, write your final answer af- ter "# #####"
Let's break it down:
* #*Fiddle Music:* Fiddle playing is a key part of Appalachian folk mu- sic. ** Appalachian Region:** The Appalachian Mountains rum through several states, but Tennessce is par- ticularly known for its strong Ap- palachian musical heritage.
Now let's alphabetize our relevant choices:
** *laska** ** 2. ** Alabbama** ** **Tnnnesee***
#### Therefore, the answer is "laskaa**
Expert Prompting
Solve the task below. Impor- tantly, write your final answer after Pro "# #
Gemini 1.5
Let's break this down:
* **The question about a state:*
We can ignore options like "orches- tra" and "violin case". **It's known for fiddle playing:** This makes think of places with strong folk mu- sie traditions. *** **Alhabettiaal order is koy: We need to focus on the first letter of each state.
You simplify complicated ideas into something easy understand. Your common sense shines clear guide, helping people when they're stuck on complex issues.
Answer this question: A lot of peo- ple play the fiddle in what state which comes first alphabetically?
Out the remaining options (Alabama, Tennessee, Alaska),
The choices are the following: 'al- abama', 'orchestra', 'tennessee', 'vio- lin case', 'alaska..
** Alaska ** comes first alphabeti- cally.
# ## Alaska
43


Reviews:
[{"review_id": "HnFXcheCuJ", "reviewer": "Reviewer_as4w", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents an extensive replication to evaluate the performance of LLMs in several benchmarks. The empirical obtained from the study shows a significant deviation from the reported results in the literature. Specifically, in the *chain-of-thoughts promting*, the replication from the paper shows no statistical difference compared to the improvement reported in previous studies. Similar observations (e.g., no significant or only a slight improvement) are also in * Sandbagging*, *EmotionPrompting*, *Re-Reading*, * Rephrase-and-Respond* and * ExpertPrompting*. Such results have, therefore, motivated the authors to make several recommendations when evaluating LLMs, including benchmarks, models, model versions (or model updates) and the outputs.", "claims_and_evidence_yn": "No", "claims_explanation": "I do appreciate the effort done by the authors to re-evaluate the performance of several LLMs on different benchmarks. Although the paper points out several inconsisent results reported in the literature, such as no improvement but was reported as significantly improved, this is unfortunately a serious-but-known issue in machine learning research in general. This issue has led to the effort of reproducibility without much success, and now, it has been resurfaced as evidenced in the paper.\n\nNevertheless, the contribution of the paper is still insufficient and simply stops at evaluating on their own benchmarks without further analyzing the causes to fix. For example, in the recommendation for the benchmarks, the paper argues about the quality, such as typos or errors, which contributes to the inadequate benchmarks. However, a controlled study, in which two sets of results: one for the one with typos and errors, and the other for the one after fixing, would give a more solid conclusion.", "audience_yn": "Yes", "audience_explanation": "At the current form, the paper, to me, is another reproducible effort to verify previous claims, rather than contributing further to the research in this subfield. The authors did argue the difference between reproducibility and replication, however, the current study claiming as \"replication\" does not reveal much insights as well as a solid or a work-around way to fix. The recommendations at the end are, even though theoretically appealing, too vague without specific actions or benchmarks for the community to follow.", "requested_changes": "The paper would be strong for a publication if the recommendations are more specific, such as a proper benchmarks for the community to use, or a way to fix the output of LLMs (similar to fixing a random seed).", "broader_impact_concerns": "The paper reports the problem in the current evaluation of LLMs using prompting techniques, and hence, impactful.", "additional_comments": "**Minor errors/comments**\n- The double quotes should be written as ``some quote'', not the one copy from other text editor as the one seen in subsection 2.1 at the bullet point \"EmotionPrompting\".\n- The citation in subsection 3.6:  B. Xu et al. (21) is inconsistent with the remaining citations in the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "xj1lQCvFwy", "reviewer": "Reviewer_sKs3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper highlights the issue of replicability in zero-shot prompt engineering techniques for Language Model Reasoning. They state that clear methodological guidelines for evaluating these techniques are lacking. The issue pointed out in the paper is crucial and of great importance to the field. In the paper, authors compare different zero-shot prompting techniques (like chain-of-thought, emotional prompting, expert prompting, etc.) on different LLMs like GPT-3.5, GPT-4o, Llama 3, etc. They created a subset of datasets (manually checked) from different reasoning benchmarks and tested different prompting techniques on different LLMs. Finally, authors also propose recommendations based on the problems of previous evaluations of prompting strategies.\n\n## Strengths -\n- The paper raises the question of replicability which most prompt based papers in LLMs lack, they highlight the probable problems and propose recommendations for better evaluations strategies.\n- The paper highlights some key problems with the evaluation of current zero-shot methods like the use of cherry picked prompts in Emotion Prompting.\n- Good recommendations are provided by the authors which is backed by thorough overview of previous prompting methods. They highlight the drawbacks of some zero shot prompt papers in their evaluations and suggest some useful recommendations.\n\n## Weaknesses -\nI have put the weakness in requested changes section.", "claims_and_evidence_yn": "No", "claims_explanation": "I am not convinced about the claims of the paper, I have put my exact questions in requested changes and will be interested in authors answers.", "audience_yn": "Yes", "audience_explanation": "Yes the problem is genuine and there is a need of better evaluation criteria, lot's of people will be interested in it.", "requested_changes": "## Weakness -\n\nI have highlighted the critical points necessary to get my recommendation by [critical] at the start of the points.\n\nI see two major weaknesses in the paper, first one is on the reliability of the scores (it could be because LLMs already use these prompting methods internally or problems in benchmark datasets) and the second one is the missing explanation of why those prompts do work in other papers. I will expand these two points and also put 1-2 minor points.\n\n- [Critical]: For reliability, one of my major concern is the limited dataset size used in the paper. For example in zero shot prompting the original paper used 12 different benchmarks and similarly for Rephrase and Respond CommonsenseQA dataset used in the paper had 12,102 data points. These dataset/benchmarks are much large compared to the limited data points picked in this paper, even though they are manually double-checked, according to me they are few in number to say something concrete.\n- [Critical]: For reliability, as pointed out by the authors closed source LLMs might be using these prompting techniques internally so why would one use those models for comparison? Even in [1] they point out something similar (for GPT-4o). In addition to this, in appendix Llama 3 (8B and 70B) shows some improvement even for 95% confidence, which should be considerable.\n- [Critical]: For reliability. Currently the data points are drawn randomly from the dataset, could there be a better way to pick ‘good’ data points from these datasets such that the sample still maintains some properties (like complexity of the questions), this would give better insights without drastically changing the datasets.\n- [Critical]: For missing explanations. I feel the reason pointed for inconsistency in the results of prompting is still limited and very general. I understand there could be a role of bad data, specific experimental setting or others but some more finer reasoning or explanation is required. Because of this missing analysis the question of enough novelty in the paper is also raised.\n    - To give an example of more finer explanations, one could look into which points in the datasets (entire dataset with ‘bad’ points) actually worked better and whether ‘bad’ points were actually responsible for the increase of scores for these prompting methods.\n- [Critical]: I have also mentioned a paper [1] which looks very relevant, I would request the authors to compare and contrast their work with the paper and cite it in their paper.\n- In my opinion the paper is predominantly text-heavy, particularly in Section 3. The inclusion of additional tables and figures would significantly improve readability and help readers better understand it.\n\n## References\n[1]\n@article{DBLP:journals/corr/abs-2411-02093,\n  publtype={informal},\n  author={Guoqing Wang and Zeyu Sun and Zhihao Gong and Sixiang Ye and Yizhou Chen and Yifan Zhao and Qingyuan Liang and Dan Hao},\n  title={Do Advanced Language Models Eliminate the Need for Prompt Engineering in Software Engineering?},\n  year={2024},\n  cdate={1704067200000},\n  journal={CoRR},\n  volume={abs/2411.02093},\n  url={https://doi.org/10.48550/arXiv.2411.02093}\n}", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WYjVefFNTA", "reviewer": "Reviewer_VSak", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper evaluates whether popular prompt engineering techniques (Zero-shot CoT Prompting, Sandbagging, EmotionPrompting, Re-Reading, Rephrase-and-Respond, ExpertPrompting) actually improve reasoning accuracy across several LLMs (GPT-3.5, GPT-4o, Claude 3 Opus, Gemini 1.5 Pro, Llama-3 8B/70B; Vicuna/BLOOM) on five reasoning benchmarks (CommonsenseQA, StrategyQA, NumGLUE, ScienceQA, CRT). \nSurprisingly, compared to prior works, the authors report mostly no statistically significant gains, except partial, model-specific improvements (e.g., Re-Reading and Rephrase-and-Respond on Llama-3 for StrategyQA). \nThe authors argue prior positive claims often rely on method choices (quality vs accuracy, cherry-picked prompts) and propose practical recommendations to strengthen replicability and statistical rigor in future prompt engineering research.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper's claim, popular prompt engineering techniques often fail to deliver accuracy gains, is supported by careful analysis.\nThe authors test six techniques across six models on five benchmarks.\nThey assess outcomes using chi-squared tests and report p-values.", "audience_yn": "Yes", "audience_explanation": "Some TMLR readers will be interested in this paper, because it challenges a prevailing narrative: popular prompt-engineering techniques reliably boost accuracy. \nShowing largely null effects across multiple models and benchmarks is intrinsically newsworthy for practitioners.", "requested_changes": "My main concern is that the paper’s depth feels limited. \nThe main message of the paper is “popular prompt engineering techniques do not help on various models and datasets”, without explaining why its results differ from prior reports. \nAs it stands, this single message feels insufficient for acceptance.\nThe contribution would be much stronger if the authors provide following two analyses:\n\n- Reproduce the original setups: Attempt to match the setups from the prior papers (same prompts, metrics, model versions, etc) to verify whether earlier gains can be recovered.\n\n- Diagnose the divergence: If those gains are reproducible in the original setups, analyze why this paper’s configuration yields different outcomes.\n\nTo deepen the work, please also discuss plausible mechanisms behind this divergence.\nTwo illustrative possibilities are:\n\n(a) Capability maturation: earlier pretrained LLMs may have lacked intrinsic reasoning ability, so prompt engineering exposed latent capacity; newer models may already leverage these behaviors, reducing the marginal benefit.\n\n(b) Provider-side prompting: modern APIs often include sophisticated system prompts (sometimes akin to instruction-following or CoT), leaving less headroom for user-level prompt tweaks.\n\nThese two are just possibilities, and there may be additional contributing factors.\nWhat matters is identifying (and, where possible, validating) at least one concrete mechanism that explains why your results diverge from prior reports. \nThis would meaningfully deepen the paper.\n\nAlso, the models used in the paper are somewhat outdated as of September 2025.\nAdding an evaluation with more recent LLMs would strengthen the paper.\n\nLastly, here are some minor mistakes.\n\n- Correct appendix references (e.g., \"Appendix 5\" $\\to$ \"Appendix E\")\n\n- In the discussion section, \"suggest that further research is necessary ...\" appears twice.", "broader_impact_concerns": "The reviewer does not identify any ethical concerns.", "additional_comments": "N/A", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)