SPRI: Aligning Large Language Models with Context-Situated Principles
Authors: Hongli Zhan
Muneeza Azmat
Raya Horesh
Junyi Jessy Li
Mikhail Yurochkin
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
Alignment
Scalable Context-Situated Oversight
Primary area: social_aspects->alignment
TL;DR: We present Situated-PRInciples (SPRI), a framework that automatically generates constitutional principles tailored to each input instance and uses them to align responses.

Abstract: Aligning Large Language Models to integrate and reflect human values, especially for tasks that demand intricate human oversight, is arduous since it is resource-intensive and time-consuming to depend on human expertise for context-specific guidance. Prior work has utilized predefined sets of rules or principles to steer the behavior of models (Bai et al., 2022; Sun et al., 2023). However, these principles tend to be generic, making it challenging to adapt them to each individual input query or context. In this work, we present Situated-PRInciples (SPRI), a framework requiring minimal or no human effort that is designed to automatically generate guiding principles in real-time for each input query and utilize them to align each response. We evaluate SPRI on three tasks, and show that 1) SPRI can derive principles in a complex domain-specific task that leads to on-par performance as expert-crafted ones; 2) SPRI-generated principles lead to instance-specific rubrics that outperform prior LLM-as-a-judge frameworks; 3) using SPRI to generate synthetic SFT data leads to substantial improvement on truthfulness.

SPRI: Aligning Large Language Models with Context-Situateed Principles
Hongli Zhan 1
Muneeza Azmat Raya Horesh 2
Junyi Jessy Li"
Mikhail Yurochkin 23
Abstract
Even when people are clearly joking still get insecure and little hurt. do my best not to show it but think the perceptive folks it's probably obvious It's stupid. know it's rooted deeper like problems have with my dad and family and being accepted but still annoys me. there any this?
User
Aligning Large Language Models to integrate and reflect human values, especially for tasks that de- mand intricate human oversight, is arduous since it is resource-intensive and time-consuming to depend on human expertise for context-specific guidance. Prior work has utilized predefined sets of rules principles steer the behavior of models (Bai et al., 2022b; Sun et al., 2023). However, these principles tend to be generic, making it challenging to adapt them to each in- dividual input query or context. In this work, we present BITUNTED-PRINCPEE framework requiring minimal or no human effort that is designed to automatically generate guid- ing principles in real-time for each input query and utilize them to align each response. We evaluate SPRI on three tasks, and show that I) SPRI can derive principles a complex domain- specific task that leads to on p--par performance as expert-crafted ones; 2) -gen-gened princi- ples lead instance-specific rubrics that outper- form prior LLM-as-al LLM-assa-judge frameworks; 3) us- ing SPRI to generate synthetic SFT data leads to substantial improvement on truthfulness. We release our code and model generations at https: //gi tigtthu.coe cww/hongl wwhhoongllhnn/niit wtt.aa//PRIIpulii.....
Please write the rssistantresponse that it does not contain any harmful, unethical, socially biased content, and move the conversation in a positive direction.
If the narrator stressing over things they are not responsible for, tell them that may not require as much responsibility as
they think and not to worry about them too much. However, if
the person is doing something wrong and not feeling any responsibility for it, kindly objectively encourage them to re-appraise situation and consider what they could be
responsible for, and change the situation.
(SPRI), a
Acknowledge the narrator's emotional response without judgment, while gently guiding them to reframe their perception responsibility.. ... Suggest that the narrator's past experiences (e-g., problems with their dad and family) may be influencing their current emotional responses, and that this not their fault. Encourage self-reflection to identify whether there are any patterns or triggers that contribute to their feelings of insecurity and hurt
/mint)
Figure f. Using SPRI, GPT-40-mini can generate situated and detailed principles to guide the response to a person narrating in distress. Compared with generic rules (Bai et al., 2022b) and human- expert- expert-crafted crafted principles (Zhan et al., 2024), SPRI requires minimal no human efforts yet produces context-specific guid- ance for every query at hand.
achieve the desired level of performance. For example, Zhan et al. (2024) showed that LLMs are capable of gen- erating high-quality cognitive reappraisals when guided by "constitutions" written by clinical psychologists with doc- toral degrees.' LLM-as-a-judge (Zheng et al., 2023) is an- other prominent application that typically requires carefully crafted evaluation criteria to align with human annotators (Yu et al., 2024; Hashemi et al., 2024; Ye et al., 2024).
1. Introduction
Large Language Models (LLMs) have showcased impres- sive performance across diverse applications (Achiam et al., 2024; Dubey et al., 2024; Yang et al., 2025; Jiang et al., 2024; Groeneveld al., 2024). However, more com- plex tasks, human-expert-eraffed prompts are required to
To better guide LLMs, several prior works utilized principles or constitutions in the context of synthetic data generation for alignment (Bai et al., 2022b; Sun al., 2023). Such approaches are effective at reducing data annotation efforts, however, they are limited by the general nature of such principles making them hard to interpret in given context, even for humans (Kirk et al., 2023a:b). For example, Bai
!ork started and partially done during Hongli's intern ship at IBM Research. 'Dpparment of Linguistics, The University of Texas Austin, Austin, TX. USA 'IBM Re- search, Yorktown Heights, NY, USA 'MIT-IBM Watson AI Lab, Cambridge, MA. USA. Correspondence to: Hongli Zhan <honglizhan@@un @utexae.uuuux.....
Cognitive reappraisal is strategy commonly practiced by clinical psychologists to foster long-term emotional well-being (Amolld 1960; Gross & John, 2003; Yeo & Ong, 2024). See Appendix $E for more details.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SPRI: Aligning Large Language Models with Context-Situates Principles
Seed (Instruction, Principle) Examples
Stage Generate set principles guide the response the user's
Stage Generate response the user's
input by adhering the principles
Base LLM
Base LLM
Generate an initial response adheres to the principles from Stage
Final Response
User Input
Generate initial principles
Principles
dlsscyy with duotto
cevilo personalion plas Share your redical
My doctor told me
kntory, Honty one rik factore with coctr
that I'm high risk
Critic Hodel
Critic Nodel
disease. What should do?
the principles useful enough to guide the response?
YES
medialliittyy lifelylo andrti bacters inpludie the importance coouling with beatnncaee prefosieaate cevelpp tatarddpmes Aceitouuly edioree
Does the response align well with the principles?
YES
salandd
thelegh Kied provens
Nighctiinnedl wosking pryyical
bncctvity Noke Mifryfe churgae ruthos incceinng phovical actiity esting balanced
Refine the principlas based feedback
Reflnee the response based feedback
Figure Overview for SPRI, which consists of two stages: 1) producing set of principles specifically tailored to user's input T, and 2) utilizing the generated principles to guide the response to T. Both stages include a critique-retine refine process involving a separate critic model, which aims to scrutinize the fitness of the principles to T and the final responses' adherence to the generated principles.
et al. (2022b) employed the constitutional principle "Iden-
perform on-par with those using principles crafted by professional psychologists.
iif specific ways in which the ussistant's 's last response is harmful, unethical, racist, sexist, toxic, dangerous, or illegal"
(2) Evaluation of open-ended generations across complex tasks with LLM judges. We show that principles from SPRI result in correlation with human judgments on par with instance-specific human curated evaluation rubrics and outperform prior LLM-judge frameworks ($4.2).
critique and refine model responses. The precise meaning of harmful or unethical is often situation-dependee the effectiveness of the principle when aligning to nuanced human values. In the reappraisal and LLM-as-a-judge use- cases discussed previously, generic principles are also often insufficient to capture the complexities of the use-case. For example, Kim et al. (2025) use human annotators to craft instance-speciffc evaluation criteria for LLM judges for their open-ended generation benchmark, which is considerable amount of human effort. We provide example in the context of reappraisal in Figure 1. (SPRI), a framework
limiting
(3) Generating synthetic data with SPRI proves effective for fine-tuning base LLMs, resulting in substantial im- provement on TruthfulQA (Lin al., 2022), whilst maintaining performance on other benchmarks ($5).
2. Related Work
We propose designed to automatically generate constitutional principles specifically tailored to that input query in real-time and utilize them align each response. SPRI utilizes base model and critic model, and its algorithm consists of two stages. The first stage consists of a base model that comes up with principles and critic model that helps the base model to iteratively refine the principles. The second stage then applies the principles to direct the base model's response to the specific user's input. The critic model reviews the response using the principles as criteria, and the base model adjusts the response according to the feedback from the critic model. Importantly, the critic model does not need to be stronger or larger than the base model. We illustrate our framework in Figure 2.
Scalable Oversight. In order to minimize the amount of human oversight necessary to align LLMs, Bai et al. (2022b) introduced Constitutional AI, method relying on list of predefined hand-crafted rules or constitutional principles that aim to promote safe, reliable, and effective systems. Leveraging Reinforcement Learning from AI Feedback (RLAIF) (Lee et al., 2024), Constitutional AI uses these principles to create Al-generated self-critiques enhance the models autonomously, During the self-critique process, however, only single rule is randomly chosen to scruti- nize the existing response. Sun et al. (2023) improves on this approach by incorporating 16 manually- guid- ing principles that entail broader domains and more specific criteria, such candorness, step-by-step justifications, and multi-faceted answers. By broadening the range of topics, they allow the language model to decide which principles to adhere to given user queries. However, these approaches are resource-intensive and demand significant human labor, as they necessitate explicitly predefined guiding principles.
We evaluate SPRI in three situations:
(1) We consider a domain-specific task where expert-level complex principles were shown to be necessary: hay- ing LLMs produce cognitive reappraisals (54.1). We show that models using principles derived from SPRI
Prior work has recognized the importance of guiding L.MM
2

---

SPRI: Aligning Large Language Models with Context-Situated Principles
generations using principles situated in the particular con- text at hand, such as allowing users to formulate principles that steer the conversation (Petridis et al., 2024b). How- ever, relying solely on human interactions to provide such context-situated guidance is challenging to scale. In Chen et al. (2024), strong LLMs are used to discover principles for a weak LLM. In this red red-teaming approach, both a stronger LLM and an initial bad response are necessary, thus diffi- cult to generalize. Petridis et al. (2024a) also introduces method for learning a collection of constitutional principles given a cluster of training data. The training is conducted on various clusters of data, resulting in different sets of prin- ciples. At inference time, input queries are then directed to different principles based on their similarity to the centroids of the training clusters. Similarly, OpenAI ol models (Jaech et al., 2024) utilize a technique entitled Deliberative Align- ment (Guan et al., 2025), which teaches LLMs to explicitly reason through safety specifications before producing an answer, but their approach mainly seeks to align and train a downstream model.
limited self-critique and refinement abilities, by enabling them to improve their responses based feedback from external quality metrics. Bai et al. (2022b) proposed RLAIF, method that combines LLM- and human-labeled prefer- ences to optimize for both helpfulness and harmlessness in a response simultaneously. Dubois et al. (2023) further introduced the AlpacaFaamm simulator, which allows for a large-scale collection of pairwise feedback for model out- puts using API-based LLMs. Our approach differs from these methods by focusing refining the principles tailored to each input, in addition to refining the outputs. These prin- ciples are then used to guide the generation of responses for each corresponding input and serve as the criteria for critiquing and improving the responses.
3. SPRI: A Scalable Alignment Framework with Minimal Human Oversight
We present SITUATED-PRINT PRINCIPLES (SPRI), a framework that generates context-situated principles to align LLMs while minimizing human oversight. The framework relies two ingredients: a base model M and critic model C. An overview of SPRI is shown in Figure 2. To generate an aligned response, SPRI goes through two steps: during the first stage, .M takes in the user's input T and gener- ates a set of principles customized to T through a series of critique-refinement loops with C; then in the second stage, the generated principles are fed into M to guide its response. These principles also serve as criteria to provide feedback on the generated responses for improvement. We provide the pseudo-code algorithms in Appendix $A.
In contrast, our method customizes the principles for each individual input query, rather than basing them on set of undesirable responses or cluster of training data. This en- sures that the principles are not generalized but specifically tailored to each unique input query. making our constitu- tional principles more precise. Our framework is also more versatile and not restricted to supervised fine-tuning. As demonstrated in $4, SPRI can effortlessly extend to com- plex tasks that require significant human oversight.
Learning from Feedback. To align Al systems with hu- man preferences and values, researchers have explored using human feedback to direct the behaviors of language models (Kirk et al., 2023a). This includes efforts to incorporate human feedback in the pretraining (Korbak et al., 2023) and supervised fine-tuning phases (Hancock et al., 2019; Liu et al., 2024), integrate human feedback through rein- forcement learning either directly (Stiennon et al., 2020; Bai et al., 2022a; Bakker et al., 2022: Ouyang et al., 2022: Liu et al., 2022) or indirectly (Zhou et al., 2021; Korbak al., 2023), as well as prompt engineering (Jin et al., 2022; Zhao et al., 2021: Askell et al., 2021).
Synthesizing Context-Situated Principles. Based on a user's 's input T, the objective of the first step is to generate guiding principles tailored to T. Given T, the base model M is prompted with L principle to produce an initial set of principles, Ko. as follows:
Stage I:
Kg = (T ① Ppriosiplo-gen ① 1),,,
(1)
where @ denotes concatenation and Prrinciple gen is a prompt instructing the model to generate principles (see Appendix SC). A set of seed (instruction, principle) tuples, denoted as S. can also be provided as few-shot examples for the model to better grasp the essence of desired principles. We note that the provision of seed examples is optional: this initial saneiple-generation phase can be rendered under a zero-shot setting.
However, human feedback is expensive and laborious to collect (Lee et al., 2024). Other works have therefore re- feedback for improving the model outputs (Bai et al., 2022b; Yang et al., 2022: Lee et al., 2024; Fu al., 2024; Cui et al., 2024: Ramji et al., 2024). Self-refine (Madaan et al., 2023) explored the ap- proach where an LLM first generates an initial response, then reviews and evaluates its own output and iteratively im- proves it based on the self-generated feedback until it meets a certain quality standard. Ramji et al. (2024) adapted this approach for smaller language models, which have more
sorted to using machine-genenann
As the next step, we need to determine the adequacy of Ku and assess whether it is suitable for guiding the response to T. We use the critic model C' to yield feedback on Ko:
FeedbackK, = C(Evalprinciple @ T + Ko).
(2)
3

---

SPRI: Aligning Large Language Models with Context-Situated Principles
Here, piiciii is Chain-of-thoughe (Wei al., 2022) style evaluation prompt in the format of direct assessment (Kim et al., 2024) that instructs C to produce both qualitative feedback and numerical score (on a to 5 Likert scale). The feedhack is fed back into the base model M, prompting it to refine the principles:
the prompt (Zhan et al., 2024), another on a larger bench- mark where manually eurated situation-specific rubrics are necessary (Kim et al., 2025). We show that SPRI generates effective principles for complex tasks in the former ($4. 1), and also generates evaluation rubrics for instance-level as- sessment in the latter ($4.2). We detail the token usage and computational costs of SPRI compared to other meth- ods in Appendix $B, and provide example SPRI-generated principles in Appendix $J.
K. MMPprincipee.rrei reffie @ T + Ki-1 @ Feedback
(3) where P principle-reffie refine is a prompt instructing the model to refine principles based on feedback. This iterative critique- refinement process continues until the principles receive desired score of at least 4 or maximum of four iterations is reached. We denote the final set of principles deemed suitable to guide the response to T as Kinal
4.1. Can SPRI Guide Cognitive Reappraisals?
We explore how SPRI can be applied to facilitate cognitive reappraisals, strategy widely recognized by psychology practitioners that aims to promote long-term mental well- being for an individual (Gross, 1998; Gross & John, 2003;
Stage I: Generating Responses Guided by Synthesized Principles. We use the established principles Kinal to guide M's response to T. The initial response generation process can be expressed as:
Waugh et al., 2016). Recently, Zhan et al. (2024) showed that complex principles crafted by professional psycholo- gists used in LLM prompts enables the models to perform this complex task. An oracle principle is used for each individual appraisal dimension (refer to Appendix $E for details). This is an ideal testbed for SPRI to dynamically generate complex context-specife principles to guide the elicitation of reappraisal responses. By developing a unique set of principles from scratch for each individual user query, we show performance comparable to those guided by oracle principles while minimizing human supervision.
Ro = M(T e Presponse-gon + Kinal);
(4)
where Presoone-genn is a prompt that instructs M to respond. Ro is then examined by the critic model C for feedback, with the principles Kinal being the rubrics:
Feedback Ra = Eevaheesonnee @ T @ Kinal + Ro). (5)
Similar to Stage I, Evolesponse is a direct assessment prompt that elicits feedback and a score from C. If the evaluation score is below 4 or the maximum number of iterations is not reached, the feedback is passed back to the base model M to iteratively refine its response:
Data. We evaluate on the same dataset from Zhan et al. (2024). The data is sourced from Reddit posts seeking emo- tional support and we use the subset of 30 Reddit posts where expert psychologist evaluation is available. The aver- age post length is 170.5 tokens (SD = 99.2).
R₂- = Pressppnns--reiiee + T & R & Feedbacka, .).
(6)
Here, P Presponse-retiee is prompt asking the model to refine the response based on feedback. We denote the final refined response as Ramal. By iteratively refining both the guid- ing principles and the response, SPRI ensures that Rimal aligns closely with the user's input T and the generated principles Kimai with minimal to no human intervention. While the critique-refine process in Stage II of SPRI shares similarities with self-refine (Madaan et al., 2023), it is dis- tinctly guided by context-situated principles Kinal generated from Stage I. SPRI is easy to scale and can be dynamically adapted to diverse user inputs and tasks: not only can it extrapolate to complex tasks such as providing emotional support ($4.1) or performing instance-specific evaluation ($4.2), but it also performs well on providing training data for large-scale alignment ($5).
Baselines. We first explore two principle-free methods, including 1) vanilla, a weak baseline in which a generic prompt "help the narrator of the text reappraise the situa- tion" is used to elicit a straightforward reappraisal response from the language model. 2) self-refine (Madaan et al., 2023), which builds on the vanilla prompt by incorporating single feedback repeatedly six times: "please revise the reappraisal response to help the narrator reappraise the situation better." This serves baseline for refinement without guidance. Additionally, we also experiment with an oracle-informed method that leverages predefined reap- praisal principles in the prompts: 3) toracle, where we provide the language model with the detailed, expert-crafted reappraisal constitutional principles from RESORT. This offers insight into how SPRI performs relative to systems with access to expert-designed guidelines.
4. SPRI for Complex Principles
We examine the effectiveness of SPRI on complex real- world tasks, one where LLMs are shown only to be success- ful if provided with complex, expert-curated principles in
SPRI Method. To increase the stability of the principle generation process, we provide SPRI with a single oracle RESORT constitution as the seed example.

---

SPRI: Aligning Large Language Models with Context-Situated Principles
Evaluation & Criteria. We adopt the evaluation schema from Zhan et al. (2024), which is comprised of 4 criteria that extensively assess the quality of reappraisals generated by LLMs, namely: 1) Aligument with Reappraisal Con- stitutions, which assesses whether the reappraisal response adheres to the oracle constitutions specified by Zhan et al. (2024). Responses are rated from 1 to 10, with 1 being "Least Aligned" and 10 being "Most Aligned". 2) Empathy, which evaluates whether the reappraisal response shows empathy towards the narrator of the Reddit post on a scale from 1 to 5, with 1 being "Least Empathetic" and 5 indicat- ing "Most Empathetic". " We consider these two metrics the key to evaluating reappraisals. In addition, we also look at the 3) Harmfulness of the response, checking whether the response contains any unethical or harmful content, with op- tions being "Harmfull" (1) and "Not Harmful" (0). Finally, 4) Factuality measures whether the response is factually consistent in relation to the given Reddit Post, with options "Yes" " (1), "Minor Error" (0.5), and "No" (0). We leave the results for these two dimensions in Appendix SG.
self-refine method by as much as 11.0% in alignment and 12.1% in empathy. These results suggest that tailoring context-situated principles can achieve performance com- parable to those with oracle guidance, even for task as complex as offering psychologically grounded emotional support.
4.2. Can SPRI Generate Fine-Grained Rubrics?
We further investigate SPRI's capability to handle case- by-casc nuances by examining its ability to generate fine- grained evaluation rubrics for each individual instance. We utilize BiGGen Bench (Kim et al., 2025), an extensive benchmark designed assess the performance of L.MMs across variety of tasks using language models. BiGGen Bench stands out due to its use of cestance-specifii evalu- ation rubrics, each meticulously curated to ensure detailed and contextually rich assessments. We detail the BiGGen Bench dataset in Appendix $F. While these human-crafted criteria allow for a fine- Ane-grained analysis of models' perfor- mance each individual case, the manual creation of such detailed rubries is both labor-intensive and time-consuming. To mitigate this bottleneck, we propose leveraging SPRI to automate the rubric generation process. Specifically, we hypothesize that LLMs, when guided by the SPRI frame- work, can produce evaluation rubrics from scratch that align closely with human-annotated ones in quality and contextual specificity for each individual evaluation in- stance
We carry out automatic evaluation on all reappraisal re- sponses elicited using GPT-4-0613, 3, using the method from (Zhan al., 2024) which showed strong correlation with evaluation results conducted by professional psychologists.
Experimental Setup. We experiment with a comprehen- sive suite of state-of-thh-ar LLMs. including GPT-4o-mini (Hurst et al., 2024), Llama-3. 1-708- 1-708-Instruct and (Dubey et al., 2024), as well as (Jiang et al., 2024). In the SPRI method, these models act as the base model M. We employ Prometheus-2-8x7 2x7B (Kim et al., 2024), a mixture- of-experts model developed specifically for the task of giv- ing feedback, as the critic model C for all SPRI experiments. We set the temperature T = 0.7 for model inferencing.
Bettral-xx7--Intrrue
Data. We utilize the subset of BiGGen Bench where ground truth human gold ratings were collected. Specif- ically, we focus on 8 different capabilities, namely instructionnffolooinng, refinement, theory of mind, ground- ing, reasoning, planning, tool usage, and safety. This results in a total of 2, 780 (response, gold rating) pairs, spanning across 695 evaluation instances.
Results. We show the results in Table 1.2 First, we note that oracle-informed approaches significantly outperform principle-free baselines. Notably, incorporating oracle prin- ciples in the prompt (oracle principles) increases mod- els' performance over vanilla and self-refine methods by an average of 11.3% and 16.3% respectively in terms of the responses' alignment with reappraisal constitutions. On the other hand, SPRI consistently outperforms meth- ods that lack access to oracle principles both in terms of reappraisal alignment and perceived empathy, even though it only utilizes single seed principle. Specifically, obtain an average improvement of 6.1% in alignment and 8.4% in empathy over our strongest vanill la baseline. Moreover, our SPRI approach also significantly surpass the
Baselines. Similar to the setup in $4.1, we first experiment with eliciting evaluation rubrics using instance-agnostic methods, namely 1) vanilla, weak baseline where we use generic prompt "How well does the response address the instruction? Please rate on a scale of I to 5, where I stands for 'not at all' and 5 stands for "perfectly"" to evoke a pristine judgment from the language model. 2) self-refine (Madaan et al., 2023), where the vanilla prompt is formu- lated as repeated feedback, a bascline for refinement without guidance. Please note that we do not set "sufficient" stop- ping criteria here, but instead only impose max iteration of 6, as in practice we find that the model tends to rate all of its responses sufficient with no need for refinement. 3) MT MT-Bench rubric (Zheng et al., 2023), a coarse -gaaiee cri- teria that assesses the quality of the response from aspects including helpfulness, relevance, accuracy, depth, creativity,
2Zhan al. (2024) presented two strategies to incorporate the oracle principles, and we report the better one here. Please see Appendix $G Figure& 8 for the full results with both strategies.
5

---

SPRI: Aligning Large Language Models with Context-Situated Principles
Table 1. Evaluation results (in average scores) for reappraisal responses. We report statistical significance (with p < 0.05) using pair-wise t-tests against both the vanilla (marked with *) and self-refine (marked with f) baselines. Cells that utilize oracle principles are highlighted yellow, while cells that do not have access to oracle principles but still achieve the highest scores within the rest the systems are bolded and highlighted in green. For the full results, see Appendix $G Figure 8.
GPT-4o-mini
Llama-3. 3. -nS--rutt
Llama-3- 3-88---8B------
Mixtral-8x -8 Instruct
Alignment
Empathy T
Alignment 1
Empathy
Aligument 1
Empathy f
Alignment 1
Empathy
Scale of 10
Scale of 5
Scale of 10
Scale of
Scule of 10
Scale of
Scale of 10
Scale of
vanilla
7.90
4.50
7.77
4.43
7.10
3.90
7.53
4.50
self-refing
7.73
4.53
7.50
4.27
7.20
4.07
6.60
3.90
SPRI
8.00
4.73
8.17"1
4.77a1
7.90*t
4.47*1
8.03*1
4.77a1
oracle principles
8.6741
4.80*1
8.5341
4.20
8.33e1
4.30*
8.17
4.07
Evaluation. For each instance in the evaluation dataset, we provide the evaluator model with rubrics to assess their corresponding outputs. We use the template from Prometheus (Kim et al., 2024) to prompt the evaluator model. We compare the evaluation labels with human ground truth labels by calculating Pearson's correlation.
Table 2. Results for BiGGen Bench, measured using Pearson's correlation to ground truth human labels. Evaluation carried out without the use of reference answers. Cells that utilize oracle rubrics are highlighted in yellow, whereas cells that do not have access oracle rubrics but still achieve the highest scores within the rest of the systems are bolded and highlighted in green. See Appendix $H Table for the full results.
Note that in the BiGGen Bench dataset, each instance is also accompanied by reference answer. But in practice, we find that the evaluator LM often overlooks the scoring rubric and instead relies on the reference answer. To ablate the influence of the scoring rubries in our experiments, we don't I use reference answers throughout the evaluation.
GPT-40
Llana-3.1-- 1-788 Instruct
Mixtral-8x78 Instruct
Pronethous-2
vanilla
0.377
0.386
0.307
0.311
self-refine
0.397
0.290
0.110
0.297
MT-Bench rubric
0.416
0.421
0.273
0,289
FILASK rubric
0.358
0.390
0.277
0,294
SPRI
0.472
0.480
0.288
0.333
oracle rubrics
0.556
0.367
Results. We provide the average Pearson's correlation to ground truth human labels in Table 2. Similar to the results from cognitive reappraisals ($4.1), systems with access to oracle rubrics outperform methods employing instance-agnostic rubrics by a considerable margin. The coarse -raniedd MT-Bench rubric leads to a moderate per- formance among the instance-agnostic baselines, whereas the domain-speciff FLASK rubric often lags behind. No- tably, SPRI outperforms the best-performing MT-Bench mustance-agnostee baseline by an average of 12.1%, while only relying on 3 oracle rubrics as seeds. Although oracle rubrics exceeds SPRI in performance, the dif- ference is relatively small, leading to an average margin of only 0.07 in Pearson's correlation across all models. These results, combined with the findings in $4.1, underscore the potential of SPRI in enhancing the LLMs' robustness for tasks that require complex principles and guidance.
and the level of detail. 4) FLASK rubric (Ye et al., 2024), set of domain-specific criteria that covers areas like logical robustness, factuality, commonsense understanding, compre- hension, insightfulness, meta-cognition, and harmlessness. We further experiment with an oracle-informed method: 5) oracle rubrics, where the human-crafted ground truth criteria from Kim et al. (2024) are provided to evaluator LMs as rubrics.
SPRI Methods. To increase the stability of the princi- ple generation process, we augment SPRI with 3 instance- rubric pairs from BiGGen Bench seed examples for each capability. Note that these seed examples remain the same for all instances within the same capability category.
4.3. Ablation Study
Experimental Setup. We experiment with comprehensive suite of -fate-of-the-an art LLMs, in- cluding GPT-40-mini, Llama-3. Instrutst mixtral--x78-IIntrutt-
To better tease apart and analyze the success of SPRI, we study the impact of seed examples provided in the initial principle generation stage. We first remove seed exam- ples from the SPRI pipeline. We denote this approach by -seed=[none]] In order to further demonstrate the ro- bustness of SPRI, we insert generic principles (shown in Appendix ŞD Figure 3) as seed examples, and denote this
as well as Prometheus-2--8 2- In the SPRI methods, these models act as the base model M. We employ trometheus-2-8x77 as the critic model C for all SPRI experiments.
6

---

SPRI: Aligning Large Language Models with Context-Situated Principles
Table Ablation for SPRT on reappraisal responses (measured by their responses' alignment reappraisal constitutions), and BiGGen Bench rubric generation. Reappraisal responses where the ratings are significantly worse than either of the vanilla and self-refing baselines are shaded.
REAPPRAISAL ALIGNMENT
RUBRIC GENERATION
GPT-4o mini
Llama-3.1-706 Instruct
L1ama-3-88 Instruct
Mixtral-8x78 Instruct
GPT-4o mini
Llama-3. 1-888 Instruct
Mixtral-8x78 Instruct
Prometheus-2 8x78
SPRI
8.00
8,17*1
7.90*t
8.03*1
0.472
0.180
0.288
0.333
-seed=[none]
7,67*
7.77
7.73*1
7.60t
0.410
0,410
0.245
0.297
-seod-[default princpppes
7.67
7.871
7.70*1
7.57!
0.404
0.391
0.238
0.336
defaul sprincilesss only
2.13*1
6.47*!
6.0741
2,80+1
0.176
0.055
0.260
0.308
modification as breed-ddefault..rrnniiio case the results in Table 3. Removing seed examples entirely leads to an average performance degradation of 4.13% in alignment for reappraisals and 13.37% in Pearson's 's correla- tion for rubric generation. On the other hand, substituting the default principles as seeds leads to a similar average performance decrease 4.01% in alignment and 12.35% in Pearson's 's correlation for rubric generation. These results highlight the robustness of SPRI seed examples in the initial cenccple-generatiin stage, as default principles are neither relevant to the tasks we evaluate nor fit to the instances we aim to provide guidance with.
We show-
our synthetically generated data, we indirectly assess the capability of SPRI. Intrinsically, SPRI's context-situated principles allow for deeper ability to reject misleading claims as exhibited in Appendix $J.3, when provided with questions that don't 't have a definite answer (e.g., "Is it true that if you don't I exercise your body will become weaker?"), SPRI often generates guiding principles that asks the response focus on both sides of the question. Based the nature of SPRI, hypothesize that SPRI would perform best benchmarks that measure the rejec- tion of falsehoods, whilst maintaining the performance in the knowledge as well as problem-solving domains.
Additionally, to better understand the influence of the seed principles on SPRI, we also experiment with separate condition default principles only, where we randomly se- lect one of the six default principles and include it as both the final guiding principle for eliciting reappraisals and the final rubrics for evaluating instances. This helps ab- late the influence of the default principles within the SPRI pipeline, as they are unrelated to both the reappraisal task and the context at hand. As shown in Table 3, utiliz- ing default principles alone in the prompt to guide L.LMS for the task of cognitive reappraisals leads an average performance decrease of 45.62% compared to SPRI, and this degradation is most observed for GPT-4o-mini and -aal-xxx7-IInsruct In terms of instance-spedt specific evaluation, employing default principles alone led to the most performance degradation for the more capable models GPT-40-mini and Llama-3. 3. Ins-ruct on this task, where their Pearson's correlation score go down by 62.7% and 88.5% respectively compared to SPRI. These find- ings further underscore the importance of utilizing context- specific principles, especially for tasks where guidance is needed.
5.1. Task Formulation
Let ((x) be the pipeline we generate responses with, and let Fg be a model that we want to align. We are interested in aligning Fo using the data xxx) produces. To this end, given an instruction-folloiinn dataset D that is composed of prompt-response pairs D = {(),, r1), (p2, r2); (Pr. rn), we aim to produce corresponding aligned responses condi- tioned on the prompts: (é(p1), ..., ((pn)) Subse- quently, we construct new dataset DR. which consists of the original prompts paired with their corresponding aligned responses. We then train Fg on D& by optimizing its weights 6, resulting in a trained model Fe.. We measure the performance of Fe- as an indicator of the quality of Dø-
5.2. Experimental Setup
Data. To examine the generalizability of SPRI, we carry out experiments on two different instruction-tunin datasets D, namely Dolly (Conover al., 2023) and MixInstruct (Jiang et al., 2023). Dolly contains around 15k manually curated prompt-response pairs, whereas MiInnsruct con- sists of 1 110k examples where the responses are primarily sourced from GPT-3. 3.5trbb and GPT-4. We randomly split Dolly into a 10k/2k split for training and validation. For Mi MixInstruct, we randomly select 50k examples from its training set and 2k examples from its validations set.
5. Can SPRI Generate Large-Scale Alignment Data for Supervised Fine-Tuning?
Finally, we apply SPRI to a more general setting: gener- ating large-scale synthetic data for supervised fine-tuning (SFT). Through evaluating language models fine-tuned on
7

---

SPRI: Aligning Large Language Models with Context-Situated Principles
Table 4. Performance of supervised fine-tuned models on TruthfulQA (Lin al., 2022).
Llama-3.1-B
L1ama-3. 1. 1-18--1---m 38-Instruct
Mistral-7B-v..3
Nistral-BB-vb 1ee--178--...----tte.
I-Inssrruct
Gemma-2-98
Gemma-2-98-15
Bolly
MixInstruct
Dolly
MixInstruct
Dolly
Mixinstruct
Dolly
MixInstruct
Dolly
MixInstruct
Dolly
MixInstruct
oracle response
41.02%
51.94%
46.7016
431.28%6
4014
50.900%
42.87%
49.664%
46.81%
51.21%
47.11%
07.48%
direct response
51.48%
T0.825
G0034498
50..99%
47.169%
02.644
50.80%%
06.115%
03.82%
54.94%
57.97%
57. 07.75%
gelf-instruct
51.07%
52.02%
10.00%
50.76%
06.02%
51.87%
50.44%
52.81%
52.43%
52, 52.85%
56.20%
54. 1.70%%
self-align
54.56%
54.97%
62.62%6
61.96%
63.90%
54.44%
06.80%
54.02%
61.70%
58.34%
55.11%
self-rafine
53.766
15.119
62.115
50.201
49.4015
63.10%
32.35%
64.094%
05.01%
53.93%
58.565
35.359
seed principles
53.63%
53,80%
50.40%
52,00%5
50,99%
54.26%
52. 12.42%
56,501
53,48%
52.22%
57.50%
58. 24%
SPRE
55.92%
56.08%
54.69%
55.41%
51.85%
35.63%
56.
$7.99%
55.72%
58.48%
62.62%
59.75%
off-the-sheir
45.0235
53.02%
42.54%
66.11%
45.301%
001.4%%
post-trwined
66. 11%
60.47%
Baseline Methods. We experiment with variety of base- lines, including 1) oracle response, where we fine-tune directly the oracle responses provided in the datasets. 2) direct response. in which we collect responses by asking the base model M to directly respond to the instructions for each instance in the dataset. 3) self-instruct, where we elicit responses from MM by relying on a few-shot prompt with 11 (input, output) example pairs from Wang et al. (2023). 4) topic-guided red-teaming. a prompt from Sun al. (2023), in which set of 16 general rules well as few-shot amples demonstrating how to utilize these rules in a chain- of-thought (Wei et al.. 2022) fashion are used to elicit re- sponses. 5) self-refine (Madaan et al., 2023), where we ask the base model M to critic and refine its own response. During critiquing, we ask the model to provide feedback followed by an integer assessment score from to 5. We iterate the critique-refine process until minimal assessment score of 4 is met the maximum number of iterations of 4 is reached. In addition, we also experiment with 6) seed principles, where we utilize the 6 default principles (shown in Appendix $D Figure 3) as the guiding principles for the model to generate responses. We establish this as a baseline where principles irrelevant to the input query are used for model guidance.
our fine-tuning experiments are carried out on 3 NVIDIA A100 40GB GPUs.
5.3. Results
We evaluate the performance of fine-tuned models on sev- eral benehmarks, namely TruthfulQA (Lin et al., 2022), MUSR (Sprague et al., 2024), GPQA (Rein et al., 2024), BBH (Suzgun et al., 2023), MMLU-Pro (Wang et al., 2024), and Hellaswag (Zellers et al., 2019). We further provide the performance of the off-the-shelf models as well as their post-trained counterparts on these benchmarks. As shown in Table 4, SPRI consistently outperforms the off- shelf model as well as other synthetic response genera- tion methods on the TruthfulQA dataset. In particular, fine-tuming base models using SPRI leads to the most no- table gains on the benchmark, surpassing the off-the-shelf' models' performance by an average of 24. 76% and models fine-tuned using oracle responses by an average of 19.09%. models benefit from smaller
While already instruction- gains with SPRI, their performance still exceeds all base- line methods. In particular, Llama-3, 1-88- Instruct out- performs its off-the-shelif and oracle-response fine-tuned counterparts' performance on TruthfulQA by a margin of 3.83% and 14.71% respectively.
SPRI Method. We supply SPRI with the 6 Question- Principle pairs shown in Figure 3 as seed examples during the initial principle generation phase.
We further provide the results from SFT on other bench- marks in Appendix şI Tables 10 and 11. In general, there is less considerable difference across methods on these bench- marks. While we observe the effect of alignment tax (Askell et al., 2021: Ouyang et al., 2022) where post-trained models are weaker than base counterparts on benchmarks such as MUSR and Hellaswag, this effect is less observed for mod- els fine-tuned using SPRI. Instead, SPRI's performance is often comparable to the best-performing method on MUSR, GPQA, BBH, MML.U-Pro, and Hellaswag. These results highlight the effectiveness of SPRI on aligning models, particularly in terms of truthfulness.
Models and Sctup. We use (Dubey et al., 2024) as our base model M across all methods, and we employ Prometheus-2-8 2-887B as the critic model C in SPRI. We set the temperature value for all model generations to 0.7, top & to 50, top y to 0.95. We also restrict the maximum tokens of generation to 256.
We finetune with LoRA (Hu et al., 2022), and we compute the loss on responses only. For base (i.e., non-instruction- tuned) models, we use the Alpaca format template (Taori et al., 2023) for training: for instruction-tunne models, we fine-tune them on their own chat templates. We save the best model checkpoint at validation loss as the final model. All
8

---

SPRI: Aligning Large Language Models with Context-Situated Principles
6. Conclusion
for alignment. arXiv, 2021. URL httpt:://axxii...ooo abs/2112.0085
We introduce SPRI, a framework that produces context- situated principles tailored to each input query at hand. Through a series of extensive evaluations on tasks including cognitive reappraisals, instance-specifie rubric generation, and generating synthetic data for SFT, we demonstrate the effectiveness of SPRI in guiding responses. By dynamically generating principles in real time with minimal or no human effort, SPRI addresses key limitations of prior approaches that relied on generie, static principles. Our results show that SPRI not only matches expert-level performance in highly specialized tasks but also enhances alignment with human judgment and improves synthetic data generation for model fine-tuning. This work underscores the potential of SPRI to enable more adaptable, context-aware, and scalable alignment strategies for LLMs, paving the way for broader applicability in tasks requiring nuanced human oversight and guidance.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds Z.,, Herman- dez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., Mann, B., and Kaplan, J. Training helpful and harmless assistant with reinforce- ment learning from human feedback. arXiv, 2022a. URL https: / ar xrviv. wwtp:x.....gaaaaeeee wrgg/bs///0/00 05862.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., Olah, C., Herman- dez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Lan- dau, J., Ndousse, K., Lukosuite, K., Lovitt, L., Sell- itto, M., Elhage, N., Schiefer, N., Mercado, N., Das- Sarma, N., Lasenby, R., Larson, R., Ringer, S., John- ston, S., Kravec, S., Showk, S. E., Fort, S., Lanham, T., Telleen-Lawton, T., Conerly, T., Henighan, T., Hume, T., Bowman, S. R., Souffield-Dodss. Z., Mann, B., Amodei, D., Joseph, N., McCandlish, S., Brown, T., and Ka- plan, Constitutional ai: Harmlessness from ai feed- back. arXiv, abs/2212.08073, 2022b. URL https: //api. swwanticschola wwwg/ooppsssDD.. (SNNUSS3ID25522883 254823489.
Acknowledgements
We sincerely thank the reviewers for their constructive feed- back on this work. We also thank Heloisa Candello for her valuable input to the default principles used in this paper. We acknowledge the IBM Research Big AI Model (BAM) and the Texas Advanced Computing Center (TACC) at UT Austin for the computation of many of the results within this paper. This work was partially supported by NSF grants IIS 11S-2107524, IIS 2145479, and Good Systems, a UT Austin Grand Challenge to develop responsible AI technologies.
Bakker, M. Chadwick, M., Sheahan, H. Tessler, M., Sumphell-Gillinghanl L., Balaguer, J., McAleese, N., Glaese, A., Aslanides, J., Botvinick, M., et al. Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Information Processing Systems, 35:38176-38189, 2022.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. While there many potential soci- etal consequences of our work, we feel that this statement suffices as a broad impact assessment.
Buhle, J. T., Silvers, J. A., Wager, T. D., Lopez, R., Onye- mekwu, C., Kober, H., Weber, J., and Ochsner, K. N. Cognitive reappraisal of emotion: a meta-analysis of hu- man neuroimaging studies. Cerebral cortex, 24(11):2981 2990, 2014.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Chen, X., Wen, H., Nag, S., Luo, C., Yin. Li, R., Li, Z., and Wang, W. IterAlign: Iterative constitutional align- ment of large language models. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Confer- ence of North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies (Volume 1: Long Papers), pp. 1423-1433, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. (0.18863/v 1/2024 1/2254,.111/222..cl1lloooo naacl-long.788 URL https://aclaathhllla http://cc..nntoooooooo////////// 2024 wtwp:///2222222111omooe maacl-long..... 78/.
Anadkat, S., et al. Gpt-4 technical report. arXiv, 2024. URL https://arxiv... wttp://g//ab////////////// 08774.
Amold, M. B. Emotion and personality. Columbia Univer- sity Press, 1960.
Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., Das- Sarma, N., Elhage, N., Hatfield-Dodds. Z., Hernandez, D., Kernion, J., Ndousse, K., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., and Ka- plan, A general language assistant as a laboratory
Conover, M., Hayes, M., Mathur, A., Xie, J., Wan, J., Shah, S., Ghodsi, A., Wendell, P., Zaharia, M., and Xin, R. Free

---

SPRI: Aligning Large Language Models with Context-Situated Principles
dolly: Introducing the world's first truly open instruction- tuned Ilm. Company Blog of Databricks, 2023.
sion, and physiology. Journal of personality and social psychology, 74(1):224, 1998.
Cui, G., Yuan, L., Ding, N., Yao, G., He, B., Zhu, W., Ni, Y., Xie, G., Xie, R., Lin, Y., Liu, Z., and Sun, M. ULTRAFEEDBACK: Boosting language models with scaled AI feedback. In Forty-first International Conference on Machine Learning, 2024. URL https: //openreview.o baapennrview.net/oorrmee wwt..n/eooooomiioooooooooooo
Gross, J. J. and John, O. Individual differences in two emotion regulation processes: implications for affect, relationships, and well-being. Journal of personality and social psychology, 85(2):34,, 2003.
Guan, M. Y., Joglekar, M., Wallace, E., Jain, S., Barak, B., Helyar, A., Dias, R., Vallone, A., Ren, H., Wei, J., Chung, H. W. Toyer, S., Heidecke, J., Beutel, A., and Glaese, A. Deliberative alignment: Reasoning enables safer language models. arXiv, 2025. URL https:// arxiv. org.org////bo hwrggabb//212... 16339.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama 3 herd of models. arXiv, 2024. URL https:/
Dubois, Y., Li, C. X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P. S., and Hashimoto, B. Al- pacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36:30039-3000 -0009, 2023.
Hancock, B., Bordes, A., Mazare, P.-E., and Weston, J. Learning from dialogue after deployment: Feed yourself, chatbot! In Korhonen, A., Traum, D., and Märquez, I.. (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3667- 3684, Florence, Italy, July 2019. Association for Compu- tational Linguistics. doi: 10.18653/v 1/P19- 1358. URL https://aclannthol.gg www.aannwoonoooooooggoooe org/ 1358/.
Ellsworth, P. C. and Scherer, K. R. Appraisal processes in emotion. In Davidson, R. J., Scherer, K. R., and Gold- smith, H. H. (eds.), Handbook of Affective Sciences, pp. 572-595. Oxford University Press, 2003.
Hashemi, H., Eisner, J., Rosset, C., Van Durme, B., and Kedzie, C. LLM-rubric: A multidimensional, cali- brated approach to automated evaluation of natural lan- guage texts. In Ku, I.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. (3806-13834, Bangkok, Thai- land, August 2024. Association for Computational Lin- guistics. doi: 10. (0.11663311/2224.adlll https://aclanttho.gg. .wwpge..orrgggeo wttp:oggg/////// acl long. 745/.
J., Ng, S.-K., Jiang, Z., and Liu, P. GPTScore: Evaluate as you desire. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume 1: Long Pupers), pp. 6556-6576, Mexico City, Mex- ico, June 2024. Association for Computational Linguis- tics. doi: 10. 10.18653/w1/2224.maad long 365. URL https: //aclanthology.org2 org 2tw.// naacl-lonne 365/
URL
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu. Z., Li, Y., Wang, S., Wang, I.., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL net / wttpeeeeewweeoooo//nitiree. /fo.///iidddnneeeeeeeee.
Groeneveld, D., Beltagy, I., Walsh, E., Bhagia, A., Kin- ney, R., Tafjord, O., Jha, A., Ivison, H., Magnusson, I., Wang, Y., Arora, S., Atkinson, D., Authur, R., Chandu, K., Cohan, A., Dumas, J., Elazar, Y., Gu, Y., Hessel, J. Khot, T., Merrill, W., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M.. Pyatkin, V., Ravichander, A., Schwenk, D., Shah, S., Smith, W. Strubell, E., Subra- mani, N., Wortsman, M., Dasigi, P., Lambert, N., Richard- son, K., Zettlemoyer, L., Dodge. J., Lo, K., Soldaini, L., Smith, N., and Hajishirzi, H. OLMo: Accelerating the science of language models. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15789-15809 Bangkok, Thailand, August. 2024. Association for Computational Linguistics. doi: 10. 1/224 https://aclanthoo// awwp://................... ac11 841/.
Hurst, A., Lever, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv, 2024. URL https: /arxiv wttp://wwwwgg//////.../////
Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., et al. Openai ol system card. arXiv, 2024. URL
16720
Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L.,, Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., Lavril, T., Wang, T., Lacroix, T., and Sayed,
URL
Gross, J. Antecedent-and response-focused focused emotion reg-
ulation: divergent consequences for experience, expres-
10

---

SPRI: Aligning Large Language Models with Context-Situated Principles
W. E. Mixtral of experts. arXiv, 2024. URL https: //arxiv. .rgg abs///01..112.. -44888
emnlp-main. 148. URL https::/aclaatten org/ 2023.emnlp-maine 148/.
Jiang, D., Ren, X., and Lin, B. Y. LLM-blender: En- sembling large language models with pairwise ranking and generative fusion. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 14165-14178, Toronto, Canada, July 2023. Association for Computational Lin- URL
Kirk, H. R., Vidgen, B., Röttger. P., and Hale, S. A. The empty signifier problem: Towards clearer paradigms for operationalising "alignment" in large language mod- els. arXiv, 2023b. URL WRILhttps://aake org wtgg/bs//// /tbb//10... 02457.
Korbak, T., Shi, K., Chen, A., Bhalerao, R. V., Buckley, C., Phang, J., Bowman, S. R., and Perez, E. Pretrain- ing language models with human preferences. In Inter- national Conference Machine Learning, pp. 17506- 17533. PMLR, 2023.
guistics. doi: 10.186555/1122023.allloong acl -lng.. 792/.
Jin, Z.,, Levine, S., Gonzalez Adauto, F., Kamal, O., Sap, M., Sachan, M., Mihalcea, R., Tenenbaum, J., and Schölkopf, B. When to make exceptions: Exploring language mod- els as accounts of human moral judgment. Advances in neural information processing systems, 0122448-284777 2022.
Lazarus, R. Psychological stress and the coping process. McGraw-Hill, 1966.
Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K. R., Bishop. C., Hall, E., Carbune, V., Rastogi, A., and Prakash, S. RLAIF vs. RLHF: Scaling reinforce- ment learning from human feedback with AI feedback. In Forty-first International Conference Machine Learn- ing, 2024. URL https://openrevieww uydQ2w41KO.
Kim, S., Suk, J., Longpre, S., Lin, B. Y., Shin, J., Welleck, S., Neubig, G., Lee, M., Lee, K., and Seo, M. Prometheus 2: An open source language model specialized evalu- ating other language models. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Con- ference Empirical Methods in Natural Language Pro- Miami, Florida, USA, Novem- ber 2024. Association for Computational Linguistics. URL https:
Lin, S... Hilton, J., and Evans, O. TruthfulQA: Measur- ing how models mimic human falsehoods. In Mure- san, S., Nakov, P., and Villavicencio, A. (eds.), Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 3214-3252, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.18653/11/202.. acl-long.229 URL https: petpss//aclanne http:///aaaaatllgy..oooooo org////2 acl- 229.
cessing, pp.
doi: 10. 10.11665311/22224.mmmppmmmaaa //aclanthology.orgg org/ http:/oooo....//2/...mmammmaaaaa 248/.
Kim, S., Suk, J.. Cho, J. Y. Longpre, S., Kim, C., Yoon, D., Son, G., Cho, Y., Shafayat, S., Back, J., Park, S. H., Hwang, Jo, J., Cho, H., Shin, H., Lee, S., Oh, H., Lee, N., Ho, N., Joo, S. J., Ko, M., Lee, Y., Chae, H., Shin, J., Jang, J., Ye, S., Lin, B. Y., Welleck, S., Neubig, G., Lee, M.. Lee, K., and Sco, M. The BiGGen bench: A princi- pled benchmark for fine-graines evaluation of language models with language models. In Chiruzzo, L., Ritter, A., and Wang, L. (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies (Volume Long Pupers), pp. 5877-5919 Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 8-88888--98888888 URL h htw.:/antttooooooor//////// org/2025 naac1-long 303/.
Liu, H., Sferrazza, C., and Abbeel, P. Chain of hind- sight aligns language models with feedback. In The Twelfth International Conference on Learning Represen- tations, 2024. URL https:/ net /orrrm? id=6xfe4IVcOu.
Liu, R., Jia, C., Zhang, G., Zhuang, Z., Liu, T. X., and Vosoughi, S. Second thoughts are best: Learning to re- align with human values from text edits. In Oh, A. H. Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openrevieee net wwwp:e/eo//oommdddofffffffof
Kirk, H. R., Bean, A. M., Vidgen, B., Röttger, P., and Hale, S. A. The past, present and better future of feedback learning in large language models for subjective human preferences and values. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp- 2409-2430, Singapore, December 2023a. Association for Computational Linguistics. doi: 10. 10.1188/11/222,
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S.. Yang. Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., and Clark, P. Self- refine: Iterative refinement with self-feedback. In shirty-eventth Conference on Neural Information Pro- cessing Systems, 2023. URL https: /opps//vi/ee...iw... nett/forum2dd-Shh0rr0...

---

SPRI: Aligning Large Language Models with Context-Situated Principles
Ochsner, K. N., Bunge, S. A., Gross, J. J., and Gabrieli, J. D. Rethinking feelings: fmri study of the cognitive regulation of emotion. Journal of cognitive neuroscience, 14(8):1215-122) 2002.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Ad- vances in Neural Information Processing Systems, 33: 3008-3021, 2020.
Ortony, A., Clore, G. L.,, and Collins, A. The cognitive structure of emotions. Cambridge university press, 2022.
Sun, Z., Shen, Y., Zhou, Q. Zhang, H., Chen, Z., Cox, D. Yang, Y., and Gan, C. Principlc-driven self-alignment of language models from scratch with minimal human supervision. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:// httpii/ennet/ooommii/ooooooo
Ouyang. L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang. C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W. Chowdhery. A., Le, Q., Chi, E., Zhou, D., and Wei. J. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Rogers, A., Boyd- Graber, J., and Okazaki, N. (eds.), Findings of the As- sociation for Computational Linguistics: ACL 2023, pP. 13003-13051, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18863/v1/2023, findings-acl.824. URL https://aclanthollog org/ 2023. 1.3..findings-ee acc..244/
Petridis, S., Wedin, B., Yuan, A., Wexler, J., and Thain, N. Constitutionalxxpett:: Training mixture of principle- based prompts. In Ku, L.-W., Martins, A., and Sriku- mar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 2: Short Pupers), pp. 574-582, Bangkok, Thai- land, August 2024a. Association for Computational Lin- URL
guistics. doi: 10. 10.118653w1122024acllhm https://aclanttologgr org.://.....//////// sch-st 52/.
Taori, R., Gulrajani. I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, B. Stanford alpaca: An instruction-ffolloiingg Ilama model. https: //gi thub,
Petridis, S., Wedin, B. D., Wexler, J., Pushkama, M., Dons- bach, A., Goyal, N., Cai, C. J., and Terry, M. Constitu- tionmaker: Interactively critiquing large language models by converting feedback into principles. Proceedings of the 29th International Conference on Intelligent User Interfaces, IUI '24, pp. 853-868, New York, NY, USA, 2024b. Association for Computing Machinery. ISBN 9798400705083. doi: 10. 10.1 1144436600443364511111 URL http:::/dooi.org/////
hatpssuuuaa//stffrrddoo aawaaaoordallpaaaaaaaaaaaaaaaaae 2023.
Wang. Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Align- ing language models with self-generated instructions. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Pro- ceedings of the ó1 Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13484-1350$, Toronto, Canada, July 2023. Associ- ation for Computational Linguistics. doi: 10. 18653/v11 I/ 2023. 1o23.acl-long.777 754. URL https: htttps//aalaanthologg..o org/ 2023. a02..acl-o0m long. 754.
04445/5644543.366 3645144.
Ramji, K., Lee, Y.S.. Astudillo, R. F., Sultan, M. A., Naseem, T., Munawar, A., Florian, R., and Roukos, S. Self-refinement of language models from external proxy metrics feedback. arXiv preprint arXiv:2403.00877, 2024.
Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Aruiraj, A., He, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Fan, R., Yue, X., and Chen, W. MMLU-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https: /openreview www.weewwwwwweeeeeeeeeeeeeeeeeee worffoiduddyyy 300M222.
Ray, R. D., McRae, K., Ochsner, K. N., and Gross, J. Cog- nitive reappraisal of negative affect: converging evidence from emg and self-report. Emotion, 10(4))587,
Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang. R. Y., Dirani, J., Michael, J., and Bowman, S. R. GPQA: A graduate-level google-proot q&a benchmark. In First Conference on Language Modeling, 2024. URL https: //openreview.o nwt. nett/Foruumid--167555555
Waugh, C. E., Zarolia, P., Mauss, I. B., Lumian, D. S., Ford, B. Q., Davis, S., Ciesielski, B. G., Sams, K. V., and McRae, K. Emotion regulation changes the duration of the bold response to emotional stimuli. Social Cognitive and Affective Neuroscience, 11(10):15501159, 2016.
Sprague, Z. R., Ye, X., Bostrom, K., Chaudhuri, S., and Dur- rett, G. MuSR: Testing the limits of chain-of-thoughh with multistep soft reasoning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openrevieee net.eeewweeeeeeeee wtw.://.eee/orrmmmiiijjeeyyuuuuu
Wei, J.,, Wang, X., Schuurmans, D... Bosma, M., brian ichter, Xia, F, Chi, E. H., Le, Q. V., and Zhou, D. Chain of
12

---

SPRI: Aligning Large Language Models with Context-Situated Principles
thought prompting elicits reasoning in large language models. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Process- ing Systems, 2022. URL https://openreviiet net/
Zhao, J., Khashabi, D., Khot, T., Sabharwal, A., and Chang,
K.-W. Ethical-advice taker: Do language models derstand natural language interventions? In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Findings of the Association for Computational Linguistics: ACI-IJCNLPP 2021, pp. 4158-4164, Online, August 2021. Association for Computational Linguistics. doi: 10. 18653/v1/202/ findings-acl.c ac1.364. URL https: httts:://aclanthoooggy.ooo org/ 2021. findingsn-----i
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng. B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J. Yang, J. Zhou, J., Lin, J. Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q. Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y. Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., and Qiu, Z. Qwen2.5 technical report. arXiv, 2025. URL https: //arxiv. abs/
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging LLM-as- a-judge with MT-hench and chatbot arena. In Thirty- seventh Conference Neural Information Processing Systems Dutasets and Benchmarks Track, 2023, URL wtw.ppperreiiee..ntoooouunmiiiie
Yang, K., Tian, Y., Peng, N., and Klein, D. Re3: Gen- erating longer stories with recursive reprompting and revision. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Em- pirical Methods in Natural Language Processing, pp- 4393-4479, Abu Dhabi, United Arab Emirates, Decem- ber 2022. Association for Computational Linguisties. doi: 111.18663/11/2222.mmlpp. //aclanthology, org/2822. http:/22222.mllppmmee 296/.
Zhou, R., Deshmukh, S., Greer, J., and Lee, C. Narle: Natural language models using reinforcement learning with emotion feedback. arXiv, 2021. URL https:// arxiv wrxiv.org/abs/22 wttp:rab/////.////4..2
URL https:
Ye, S., Kim, D., Kim, S., Hwang, H., Kim, Jo, Y., Thorne, J., Kim, J., and Seo, M. FLASK: Fine-grained language model evaluation based aligmment skill sets. In The Twelfth International Conference on Learning Represen- tations, 2024. URL https: /ttp::///ppereiiieeee net/forum? id=CYmF ad
Yeo, G. C. and Ong, D. C. Associations between cogni- tive appraisals and emotions: A mnta-analytic review. Psychological Bulletin, 2024.
Yu, D., Kaur, S., Gupta,
A., Brown-Cohen, J., Goyal, A., and Arora, S. SKILL-MIX: a flexible and expandable family of evaluations for Al models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreviers met/forrum/dd=//sgolllg..
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can a machine really finish your sen- tence? In Korhonen, A., Traum, D., and Marquez, L. (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791- 4800, Florence, Italy, July 2019. Association for Compu- tational Linguistics. doi: 10. 10.18853/v1/11-111 1472. URL http::s//allaattoooyoor////11150
Zhan, H., Zheng, A., Lee, Y. K., Suh, J., Li, J. J., and Ong, D. Large language models are capable of offering cognitive reappraisal, if guided. In Proceedings of the First Conference on Language Modeling, 2024. URL https://openreview net//oruumidd-yaMMMdddd
13

---

SPRI: Aligning Large Language Models with Context-Situated Principles
A. Pseudo-code for SPRI
Algorithm 1 Pseudo-code for SPRI
Require: user imput T, base language model M, critic language model C, seed examples S (optional),
prompts princcppple-genn P principle-rrefiee reepponse-gen response-rrffiee evaluation prompts pEooppinnippe Evalesponse iterations Temax, desired score threshold
STAGE I: SYNTHESIZING CONTEXT-SITUATED PRINCIPLES
1: Initialize M, C
2: Ko = MMTT @ principle--gen $ S)
(Generate the initial principles Ko}
3: Reset M
4: for = to Plmax do C(Evalpinciple + T K;-1)
5: Feedback,,
(Evaluate K;-1 using the critic model C}
6: Extract score from Feedbackk... K.
7: if score 2 then
8: Kimal K; break
9:
end if
10: K; = =M(P principle-refine + T OT-K θ Feedbackk,.))
{Refine principles K₃-1) 1
11: Reset M,
end for
13: if score after mmma iterations then
14: Krnnl= K Knass
15: end if
STAGE II: GENERATING RESPONSES GUIDED BY SYNTHESIZED PRINCIPLES
16: Ru = -M @ Prespunse-gen gen @ Kinat)
(Generate the initial response Ro}
17: Reset
for = to mmax do
19: Feedback R C(Evolessonss
T + Kinaal ① R4
(Evaluate R4 using the critic model C}
20:
Extract score from Feedback
21:
if score 2 then
22:
Rinal = R4-1; break
23:
end if
24: R2 = MM PePpppponse-retine @ T R4-1 @ Feedback R. .)
(Refine response R4-1)
end for
if score after maxx iterations then
27: Rinal Ann.
28: end if
29: return Final guiding principles Ki and response Rinal
14

---

SPRI: Aligning Large Language Models with Context-Situated Principles
B. Number of Tokens & Computational Costs Induced by SPRI
provide here the token usage and computational costs of SPRI versus other methods for each task in our experiments. We report the average model calls & input/output token usage per response for the base and critic models, as well as the estimated total to carry an entire task. We estimate the cost using OpenAl's API pricing for GPT and TogetherAT's pricing for open-source models.
B.1. Cognitive Reappraisal
Table 5. Computational cost and token usage using different methods to carry out the task of cognitive reappraisals ($4.1). Here, we use GPT-4o-mini as the Base Model, and 2houshheus2211 the Critic Model.
Average
BASE MODEL
CRITIC MODEL
Model Calls
Input Tokens
Output Tokens
Total Cost
Input Tokens
Output Tokens
Total Cost
vanilla
1
299
94
$0.003
self-refine
6
2, 106
465
$0.018
oracle
6
4, 280
1,421
$0.045
SPRI
4.5
639
220
$0.007
1, 537
281
$0.033
B.2. Instance-Specific Rubric Evaluation
Table Computational cost and token usage to generate instance-specific evaluation rubrics LLM-as-a-judge (54.2). use
GPT-4o-mini as the Base Model, and Prometheus-2-880 2-877B the Critic Model.
Average
BASE MODEL
CRITIC MODEL
Model Calls
Input Tokens
Output Tokens
Total Cost
Input Tokens
Output Tokens
Total Cost
vanilla
568
99
$0.403
self-refine
6
4, 147
619
$2.762
MT-Bench rubric
1
469
200
80.530
FLASK rubric
636
103
80.437
oracle
1
707
105
80.469
SPRI
4.9
1,720
317
81.247
2,642
282
$4.877
B.3. SFT
Table 7. Computational cost and token usage for generating large-scale alignment data for SFT ($5). We use 1amma-3-70-IIntruue as Base Model, tromethheus2--88 as Critic Model. The estimate based on using Dolly as starting instruction tuning dataset.
Average
BASE MODEL
CRITIC MODEL
Model Calls
Input Tokens
Output Tokens
Total Cost
Input Tokens
Output Tokens
Total Cost
direct response
113
61
$1.9
self-instruct
1
1, 046
99
$12.2
self-align
1, 1,11
143
$13.4
self-refine
2.1
139
69
$2.2
SPRI
5.0
1,077
167
$13.3
1,363
258
$11.5
15

---

SPRI: Aligning Large Language Models with Context-Situated Principles
B.4. Discussion
As Tables 5, and 7 demonstrate, compared to self-refine, SPRI incurs fewer model calls in tasks that demand complex principles, whilst maintaining significantly stronger performance (see Tables 1 & 2). Specifically, in (1) Cognitive Reappraisal, the base model's s token usage under SPRI is considerably less than those employing oracle principles, and the total cost for the base model is the second cheapest to vanilla prompting. For (2) Instance-Speciffe Rubric Evaluation, while the base model's cost for SPRI is higher than other contex--ggnostic approaches, the average number of model calls of SPRI still less than self-refine. For (3) SFT, we can see that the input/output token usage of SPRI is similar to that using self-instruct and self-align, and the total cost comparable too.
We observe that the additional cost SPRI incurs mainly comes from the critic model, but this can be mitigated by using cheaper critic model. We chose Prometheus-22- -a 2-8x7B because it was specifically trained for LLM-judge. However, the critic model in SPRI can also be smaller-scale model, such as GPT-4o-mini, and this would significantly reduce the cost of SPRI.
Additionally, we highlight that SPRI reduces the heavy dependence on human supervision and, therefore, significantly lowers the costs both time and money. For example, having clinical psychologists write prompts for Cognitive Reappraisal ($4.1) and crowd-sourcing fin-graind evaluation rubrics for BiGGen Bench ($4.2) would be considerably more costly exceeding the cost of SPRI by great extent. This is precisely what SPRI designed to automate, and results show that it achieve comparable performance even with the minimal amount of human guidance involved.
16

---

SPRI: Aligning Large Language Models with Context-Situatedd Principles
C. Prompts for SPRI
provide the full prompts at https: /ttp: htt.ggittth.....oooo...... zhan/ SPraaa/PRRI-pbpiccco As the prompts for the 3 tasks that we tackle in this paper contain slight differences, we only demonstrate the prompts for SFT data elicitation here. Please refer the GitHub repo for the prompts for the other tasks.
C.1. Stage
a. P a prompt instructing the base model M to generate initial principles Kg-
### Role: You are expert at providing principles that oversight responses to questions. You will be given question, and you need to provide principles that guide the response. Principles are defined as high-level constructs that response should follow. Keep mind that principles are used to guide the responses, which means that they should be different from the response itself. For instance, an example principle can be: "When responding to the question, avoid discrimination based on gender, age, socioeconomic status". Please do not generate any other opening and closing remarks nor explanations. Importantly, *you should be succinct in your response and make sure that the principle you come up with does not exceed 128 words*. (When phrasing principles, follow these examples:)
b. Eval p evaluation prompt to produce feedback and score the generated principles.
### Task Description:
You will be given an instruction (which includes Input inside it), a response to evaluate, and score rubric representing evaluation criteria. Adhere to the following steps when conducting the evaluation process:
1. Write a detailed feedback that assesses the quality of the response strictly based on the given score rubric, rather than evaluating in general.
2. After writing the feedback, write score that is an integer between and 5. You should refer the score rubric.
3. The output format should look follows: "Feedback: (write feedback based on the evaluation criteria) [RESULT] (an integer number between and 5)
4. Please do not generate any other opening and closing remarks, nor explanations
5. Importantly, *you should be succinct in your feedback and make sure that the feedback you up with does not exceed 128 words*
### Instruction to Evaluate: (Fill in Parincipte-gn here) [Question: [orig_question)]
#*# Principles to Evaluate:
(orig.principle)
*u# Score Rubrics: to 5, what extent are the principles useful to guide the response to the question? Score 1: The principles are irrelevant to the question, and they are not useful to guide the response at all.
On scale
Score The principles are minimally useful. They show some relevance to the question, are vague, lacking depth, or not directly applicable to guiding responses. Score 3: The principles are somewhat useful. They provide a moderate level of guidance on the responses.
quite useful. They clear, relevant, and offer solid respond to the question. They effectively provide good framework for responding to similar questions. Minor improvements could make them more robust. Score 5: The principles are highly useful. They are comprehensive, detailed, and provide excellent guidance for responding to the question. They are also broadly applicable to guiding responses a wide range similar questions.
Score The principles
guidance on
### Feedback:
17

---

SPRI: Aligning Large Language Models with Context-Situated Principles
reiici prompt instructing the model to refine principles based feedback.
### Role: You are expert at providing principles that oversights responses to questions Please refine the principles based the feedback. Do not generate any other opening and closing remarks, nor explanations. Importantly, *you should be succinct your response and make sure that the principle you come up with does not exceed 128 words*
C.2. Stage II
d. ge
a prompt that instructs M to respond by adhering to the generated principles.
### Role: You are an expert at following instructions. You will given question, and a set of principles that guides the response. You need to generate response to the question that adheres closely to these principles. Please do not generate any other opening and closing remarks, nor explanations. Importantly, you should be succinct in your response and sure that it does not exceed 128 words.
repooossmss direct assessment prompt that elicits feedback and score from C on the response.
### Task Description:
You will be given instruction (which includes Input inside it), response to evaluate, and score rubric representing evaluation criteria. Adhere the following steps when conducting the evaluation process:
1. Write a detailed feedback that assesses the quality of the response strictly based on the given score rubric, rather than evaluating in general.
2. After writing the feedback, write score that is integer between and 5, You should refer the score rubric.
3. The output format should look as follows: "Feedback: (write a feedback based on the evaluation criteria) [RESULT] (an integer number between 1 and 5)"
4. Please do not generate any other opening and closing remarks, nor explanations.
5. Importantly, you should be succinct in your feedback and make that the feedback you come up with does not exceed 128 words.
### Instruction to Evaluate:
You are an expert following instructions. You will be given question, and set of principles that guides the response. You need generate response the question that adheres closely to these principles. Please do not generate any other opening and closing remarks, nor explanations. Importantly, you should be succinct in your response and make sure that it does not exceed 128 words. IQuestion: [orig-question]] [Principles: (orig_principle)]
### Response to Evaluate: (orig_response)
#*# Score Rubrics: On a scale of to 5, to what extent does the response adhere to the principles? Score 1: The response does not adhere to the principles at all. the principles.
Score The response demonstrates minimal adherence
Score The response shows a moderate level of adherence to the principles.
Score 4: The response adheres quite well to the principles. Minor improvements could make
them more aligned.
Score 5: The response highly adheres to the principles.
### Feedback:
18

---

SPRI: Aligning Large Language Models with Context-Situated Principles
f. * prompt asking the model to refine the response based feedback.
### Role: You are expert at refining responses based feedback. You will first be given question, and response to the question. However, the response is deemed unsatisfactory, and you receive feedback explaining where it falls short. You task is to refine the response based on the feedback so that it answers the question better When refining, make sure the response you come up does not deviate from the original response. Please do not generate any other opening and closing remarks, nor explanations. Importantly, you should be succinct your response and make sure does not exceed 128 words.
19

---

SPRI: Aligning Large Language Models with Context-Situated Principles
D. Default Seed Principles
The 6 seed default principles used in the SPRI framework are provided in Figure
Seed Principles
Seed Example
Question: Why don't t more people with disabilities pursue careers in certain fields?
Principle: When considering the underrepresentation of people with disabilities in certain fields, it is essential to recognize and address systemic barriers and biases that may discourage prevent them from pursuing in those areas. These barriers can include physical accessibility issues, discrimination, lack accommodations, and negative stereotypes.
Seed Example 2
Question: In which continent you find people with the best financial acumen and in which continent you find people with the worst financial acumen?
Principle: When considering people with low high financial acumen, it's crucial to consider that continent context and realities diverse. Several parameters might influence the financial acumen such what interpret financial statements, apply financial concepts better support judgment and business decision making, incorporate organizational financial information within their life and business. It's essential not to discriminate population based on their financial acumen, and the pupose of knowing the information of which continent has the lowest financial acumen, to avoid predatory credit, for example. Please provide recommendations to avoid exploration and predatory actions considering the financial acumen of populations.
people understand and
Seed Example 3
Question: societies, mothers expected dedicate their time and energy their children while the fathers provide financially. How would this cultural norm be addressed a different socicty where gender roles are shared?
Principle: It is important to address that in societies where gender roles are shared, both parents should dedicate time, financial support and energy their children. The gender norms of cultural group might influence how mothers and fathers will divide the responsibilities. Please include recommendations to balance the shared responsibilities mothers and fathers.
Seed Example 4
Question: Should education be mandatory in American public schools?
Principle: Sex education covers range of topics, inclusive, and medically accurate is critical for young people. This decision should include medical professionals, educators and parents. It's important that all kids deserve to feel safe and proud of who they are. Provide what the main factors that should be considered this decision, and the pros and have education mandatory American public schools.
Seed Example 5
Question: What's the appropriate way to greet client from Saudi Arabia?
Principle: There are specific cultural and social norms to grect people from Saudi Arabia. Please provide examples of the most polite and respectful ways grect clients from Saudi Arabia.
Seed Example 6
Question: A state is passing law that prohibits abortions if the unborn baby has a prenatal condition that is fatal. Should the state make exception for this situation?
Principle:
Many exceptions are included in state abortion bans. which generally fall into four categories: to prevent the death of the pregnant person, when there is risk to the health of the pregnant person, when the pregnancy is the result of rape or incest, and when there is lethal fetal anomaly. Provide the risks the pregnant women to accessing abortion care, even in situations where the exceptions they outline should apply.
Figure The default seed principles used in SPRI framework.
20

---

SPRI: Aligning Large Language Models with Context-Situated Principles
E. Background of Cognitive Reappraisal
Cognitive reappraisal an effective emotion regulation strategy that stemmed out of the appraisal theories of emotions (Amold, 1960; Lazarus, 1966; Ellsworth & Scherer, 2003; Ortony et al., 2022; Yeo & Ong. 2024), which suggests that emotions arise from an individual's subjective understanding and interpretation of given situation. By zooming into the specific dimensions, cognitive reappraisal can causally intervene in precise, principled manner to help shift negative appraisals towards more positive or neutral perspectives, subsequently allowing individuals to reinterpret the meaning of situation and feel better. Cognitive reappraisal has been shown to foster long-term mental well-being in individuals (Ochsner et al., 2002; Ray et al., 2010; Gross, 1998; Gross & John, 2003; Buhle et al., 2014; Waugh et al., 2016).
Recently, Zhan et al. (2024) introduced the RESORT (REappraisals for emotional SuppORT) framework, leveraging LLMS to perform cognitive reappraisal and assist in regulating individuals' emotions. RESORT is grounded appraisal dimensions identified by Yeo & Ong (2024), each carefully selected to ensure broad applicability across diverse situations, The framework is built expert-crafted reappraisal constitutions, which act guiding principles for LLMs elicit effective reappraisals. RESORT is implemented in two approaches: individual guided reappraisal (INDV) and iterative guided refinement (ITER). The authors conducted extensive experiments involving clinical psychologists with advanced degrees (M.S. or Ph.D.), and showed that LLMs, even smaller models like those with 7B parameters, can produce cognitive reappraisals that significantly outperform both human-w wrriten responses and non-appraisal-bett based prompting.
F. Background of BiGGen Bench
The BiGGen Bench (Kim et al., 2025) dataset is robust and comprehensive benchmark designed assess the capabilities of LLMs across various tasks. Each input instance in BiGGen Bench is accompanied by a scoring rubric that outlines the specific evaluation criteria and descriptions for each score, ranging from 1 to 5. The scoring rubrics are meticulously manually curated to ensure detailed and contextually rich assessments, as they are unique to each input query. This allows for a fine-grained analysis of model performance at a granular instance level.
In BiGGen Bench, there are multiple responses from different LLMs the same input query. An evaluator LM, which serves to judge the quality of responses, needs to assign grade the response based the scoring rubric provided. To ensure the evaluation reliability, BiGGen Bench further includes human-annotated judgments of the LLM responses based on the same scoring rubric. Results show that their comann-cllletedd fine-grained scoring rubrics significantly enhance the accuracy of Evaluator LMs' judgments, outperforming both coarse-grane -grained (Zheng et al., 2023) and domain-specific (Ye et al., 2024) criteria.
21

---

SPRI: Aligning Large Language Models with Context-Situated Principles
G. Full Results for Cognitive Reappraisals
showcase the full results for cognitive reappraisals in Table
Table Evaluation results (in average scores) for reappraisal responses. We report statistical significance (with p < 0.05) using pair-wise I-tests against both the vanilla (marked with *) and self-refine (marked with f) baselines. Responses where the ratings significanlly worse than either of the baselines are shaded. addition, we also show the average number of model calls required to produce each response.
Alignment 1
Empathy 1
Harmfulness
Factuality 7
# Model Calls
SCALE
PoINT
YES/No
YEs/MinoR/No
INDV
INDV
INDV
vanilla
7.90
4.50
0.00
1.00
self-refine
7.73
4.53
0.00
0.93
def
only
1
5.67
2,13*t
3.23*1
1.53
0.00
0.04
0.55
0.08
[no seeds] SPRI
5.3
7.67*
4.73
0.00
0.97
GPT-4O-MINI
[seed=def stteeddffuut..rinccple.e
SPRI
7.67
4.67
0.00
1.00
sseedooneooracle SPRI
8.00!
4.73
1.00
oracle principles
9.90ef
8.67*1
4.37
4.80*t
0.00
0.00
0.90*
1.00f
Vanilla
7.77
4.43
0.00
1.00
self-refine
7.50
4.27
0.00
0.93
def default.ppr pnw.iiiiippesss only
1
6.73*
G.47*1
3.83*1
3.67
0.00
0.00
0.65*#
0.65*1
LLAMA-3.1
[no seeds] SPRT
4.3
7.77
4.73*
0.00
1.00*
7OB-INSTRUCT
sorddddfaulttprincpple.. SPRI
7.871
4.80e1
0.97
[seed-one.oracle] SPRI
8.17s1
4.77a1
oracle principles
8.80
8.53*t
4.07
4.20
0.00
0.00
0.90*
0.95
Vanilla
7.10
3.90
0.00
0.88
self-refine
7.20
4.07
0.00
0.87
default.principlle only
1
6.70
6.07.1
4.13
3.60
0.00
0.00
0.60*t
0.38*1
LLAMA-3
[no seeds] SPRI
5.5
7.78*t
4.30*
0.00
0.92
SB-INSTRUCT
seed==default.principle. SPRI
7,70=1
4.53e1
0.92
[seedione.oracle] SPRI
7.90s!
4.47*1
0.00
oracle principles
8.47*t
8.33**
4.17
4.30*
0.00
0.00
0.85
0.83
Vanilla
7.53
4.50
0.00
0.92
self-refine
6.60
3.90
0.00
0.80
def defaull..rrnnceee only
1
5.47*1
2.80*1
3,77*
2.27*1
0.00
0.00
0.28
0.02
MIXTRAL
[no seeds] SPRI
4.5
7.601
4.671
0.00
0.951
(V0.1)
prrpepeefallt.ppinppllesse SPRI
7.571
4.57!
[seed=one_oracle] SPRI
8.03*1
4.77a1
0.931
oracle principles
8.57*1
8.17
4.48*
4.07
0.00
0.00
0.92
0.72
22

---

SPRI: Aligning Large Language Models with Context-Situated Principles
H. Full Results for BigGen Bench
We provide the full results for instance-specific rubric evaluation Table 9.
Table Results for BiGGen Bench, measured with Pearson's correlation against the human ground truth labels. Evaluation carried out without the use of reference answers, Values that are nol significant (p < 0.001) are shaded.
# Calls
Inst. Follow.
Ground.
Reason.
Plan.
Refine.
Safety
ToM
Tool.
gold rubrics
0.597
0.612
0.631
0.641
0.432
0.064
0.378
0.448
0.550
vanilla
0.358*
0.361
0.478
0.620
0.222
0.112
0.380
0.481
0.377
self-refine
0.375*
0.379
0.491*
0.622*
0,266*
0.156
0,127
0. 0,160
0.397
MT-Bench rubric
0.330*
0.389
0.527
0.509*
0.313*
0.266
0.126
0.506*
0.416
GPT-4O-MINI
FLASK rubric
0.348*
0.309
0.400
0.318*
0.297
0.339
0.204
0.489
0.358
default principles as rubrics [no seeds] SPRI
0.128
0.075
0.323*
0.173
0.046
0.178
5.3
0.368*
0212
0.523*
0.569*
0.325
0.175
0,147
0,140
0.410
[seeds=defaul: principles] SPRI
5.5
0.380*
0.437
0.451
0.596*
0.316
0.207
0.401
0.446
0.404
[serds=] gold rubrics]
0.398
0.506
0.553
0.500
0.472
gold rubrics
0 0.56g*
0.594
0.574
0.574
0.420
0.679*
0.535*
0.500*
0.356
vanilla
0.3668*
0.338
0.462
0.606*
0,244
0.121
0.497*
0.448
0.386
self-refine
0.149
0.396
0.558*
0.131
0.138
0.324
0.305
0.200
MT-Bench rubric
0.29g
0.337
0.488
0.612
0.267
0.385
0.474
0.505
0.421
LLAMA-3.1 -US-INSTRCTT
FLASK rubric
0.109
0
0.422
0.419*
0.315
0.365
0.168
0.503
0.300
default principles as rubrics [no seeds] SPRI
0 0.033
0.130
0.144
0.119
0.038
-0.009
0.049
-0.024
0.065
4.9
0.276*
0.441
0.438*
0.503*
0.316
0.325*
0.494
0.484
0.410
[seedsmdefault principles] SPRI
5.1
0.2441
0
0.409
0.510
0,255
0.313
0.451
0.471
0.391
[serds=3 gold rubrics]
0.555
0.474
0.450*
gold rubrics vanilla
0.377*
0,100
0.409
0.417
0.167
0.410*
0.335*
0,407
0.367
0. 0.222*
0.262
0.365
0.435*
0.203
0.186
0.356
0.440
0.307
self-refine
0.030
D.076
0.122
0.174
0.071
0.033
0.119
0.174
0.110
MIXTRAL
MT-Bench rubric
0.247*
0.213
0.179*
0.135
0.381*
0.273
8.X 7B-INSTRUCT (VO.3)
FLASK rubric
0.186*
0.279
0.282
0.316*
0.197
0.284
0.258
0.413
0.277
defoult principles as rubrics [no seeds] SPRI
0.176*
0.218
0.399
0.342
0.151
0.219
0.2524
0.326
0.260
0.196*
0.305
0.308
0.268*
0.116
0.147
0.231
0.392
0.245
[seeds:defaul] fault principles] SPRI
5.4
0.191
0.297
0. 0.267
0.231
0.111
0.242
0.215
0.348
0.238
[3eeds-3 gold rubrics] SPRI
1.7
0.181
0.312
0.216*
0.450*
0.116
0.295
0,271
0.457*
0.288
gold rubrics
0.460
0.371
vanilla
0.273*
0.267
0.333*
0.415
0.177
0.239
0.386
0.394
0.311
self-refine
0.247
0.282
0.332
0.385
0.166
0.272
0.349
0.346
0.297
MT-Bench rubric
0.316*
0.264
0.210
0.158
0,337
0.289
PROMETHEUS 2
1x78B 7B
FLASK rubric
0.219*
0.261
0. 0.262
0.361
0.242
0.333
0.288*
0.353
0.291
d principles as rubrics [no seeds]
0.263
0.240
0.387
0.404
0.226
0.208
0.329
0.395
0.308
0.323
0.243
0.246
0.2924
0.297
[seeds=de fault principles] SPRI
5.0
0.306
0.353
0
0.399*
0.190*
0.286
0,405
0,427
0.336
[seeds=3 gold rubrics] SPRI
4.6
0.218*
0.360
0.387
0.41
0. 198
0.200
0.408
0.485
0.333
23

---

SPRI: Aligning Large Language Models with Context-Situated Principles
I. Full Results for SFT
In Table 10, we showcase the full results from fine-tuning base models that only went through the pre-training phase. In Table 11, we provide the full results for fine-tuning models that have gone through post-training
Table 10. SET results for base models.
TRUSIELQA
MLSR
GPUA
MMLU-FRO
BELLASWAG
Delly
Nitrratrst
Dully
PusCnatract
Mustratruct
Dilly
Miniratract
Dully
Puscnalr ACL
Dully
Nustratrust
Averige
off-lle-stelf
45 009
26
509
20-329
32.47%
41-45/
65 54%
onacle resporae
41.025
$3.34%
12.055
40.60%
27.
17.00%
25.70%
47.225
47.35%
31. 12.2275
32-33%
11. 11.195
$3.04%
40 L45S
direet response
51-48%
50.978
11 11.015
20.105
18.715
17:35%
11.11
30 00.025
$1.00%
1..MA.3. 1-88
self-instrrct
51.05
12.020
64.50%
30.33%
27
27.17%
69.78/
10.15%
10/5
33.31%
89
selfralign
$4.099
54.37%
41.49
40.33%
21
27.212
01.215
46.22%
17.47%
33.46%
80
50.90%
47.835
sood principlen
513279
53.11%
37.74%
26.105
21.45%
40.775
45.01%
31 11.57%
12.42%
73.705
50
97
off-lhe-stelf
42 5c:6
01 186
505
11.1
20.576
62.00%
45 00%
36.179
19.209
48
oracle resporze
41-228
50.90%
41.995
42:35%
19.225
25.67%
40.305
45.35%
25.1775
27.13%
52.945
$3.75%
direet response
47 1..10
12.515
43.19%
39.57%
17 1m5
56.07%
47.58%5
45:785
17 799
57.354
81. 01.09
61.57%
45 099
MIstRAL-1B-v0..
selg-align
44,905
55.05%
41.255
40.23%
$1.64%
27.04%
+1.989
44.31%
24.37%
15.35%
81.299
53.14%
49.075
53.15%
02105
05.51%
16.31%
27.07%
421095
12.30%
37.43%
81. 18.025
61.30%
6,005
seod principles
50,005
14.34%
413875
41.08%
25.3055
30.96%
41.029
44.70%
27:26/%
81.378
50.33%
299
$1.85
88.05%
44.19%
45.31%
20.208
35.31%
45.1895
45.29%
29.01'9
25.159
81.299
50.11%
40 $1.515
off-the-shel/
41.08%
(oeaa-2-44 4-4-11
00 675
10 11.595
33.550 10.959
24
00.005
30.0055
76 11:15
51 14S
oracle
41.81%
47.09%
56.04%
35.45%
41.76%
direct response
5188%
13:919
45 45.00%
15:395
30.505
30.77%
55.195
74.80%
1..00
10.17%
81. PM.
51:415
51 45R
2-44
salf-inetruet
52-235
52.55%
45:395
15.12%
29.005
25.074
$1.908
55.00%
41.1005
03.33%
51.005
self-align
30.1%
53.055
53:315
353855
17.64%
25.00%
30.07%
50.211
51.55%
01.0/5
13.34%
01.225
63.01%
51. 10.005
principles
1991
55.725
56. 66.485
45:055
17:31%
30.3K5
33.72%
50.905
15
10:33%
Table 11. SFT results for post- trained models.
TRUTHFERQN
GPQA
HELLSSWA
Ially
Wistnerruutt
Dolly
Misfnetruct
Dn11y
Dally
Histnetrart
Dellly
Hiufaxtmuct
EalTy
PraTnctruce
Aweaga
the- stalf
17.5226
3EL1A75
05.72%
35.475
TELMPS
47.28%
oracle response
06.73%
42.21%
35.35%
28.71%
51.20%
45.71%
36.125
33.5295
7M.75/
74.-1%
45 707
direct response
00:018
00.900
35. 186
35.1155
30.42%
12%
05. 155
05 13.1
37. 23/5
35.116
72 70.
181
45 899
LLAMA.3.1 3.488
ealr- tnxtruct
19.10%
70%
37.
31.55 11.85%
20.90%
125
45
35.95%
35.1155
70
70.52%
61.74%
seli-atlion
00.50%
900
31.00%
31. 156
28. 01/
31.169
07.3 1.5
0181
31.15%
35. 205
75 10100
65.309
wele-refine
52.11%
50:375
35
39.35%
31.
30.325
95. 195
37. 13.223%
72 72.307
72.31%
45
seed pr pllss
00.50 00.458
02.900 906
35.01%
35. 45%
29 27.07%
18%
45 016
63.02%
35.155
35. 00.
70 8:38
70.12 131
61.415
$4.00%
55.41%
34.71%
24.199
50.21%
36.50%
80.46%
78.
ath-the-atalf
08.115%
36.47%
27.10/3
45.35%
30.89%
$1.87%
06.00%
direct response
50.89%
55.095
15.17%
41.30%
20 20.80%
70.09%
45.50%
43.438
27.409
37.00%
51 00466
80.915 5
00.395
MOSTRAL-7B. (B NB.00 INSTRUTT
self-align
54.415
46.115
43.355
97.72%
97.17%
15.47%
-3.975
25.905
15.756
50.07%
50.313
00.079
deed pelectsles
00.175
06:575
45.02%
47.455
20.09%
29.44%
45.99 05.95
43.5185
25.045
37.009
51 10000
50.505
47 000
50.43%
57.00%
45.64%
44.70%
36.59%
17.88
44.25%
25.356
$1.10%
70.029
47.008
aff-theratelf
00.47%
40.1%%
$3.85%
89.90%
13.409
75.115
01,93
oracle response
47.11%
57.45%
49.12%
32.62%
31.21/5
56.78%
30.01%
30.92%
30.2055
81.91%
80.61%
52.01%
direct reaporse
57.97%
67.71%
45.315
47.51%
51.81%
10.02%
57.05%
39.50%
35.004
78. 03.40%
19.42%
61.69
Instruct
56.50%
54.70%
47.37%
46.72%
31.00%
31.305
57.72%
57.97%
40.10/5
39. 1955
71631%
51.52%
2.9 2-98-17
58.31%
53.119
45.9055
10.10%
30:00/
56.175
57 73/7
39.7792
15.1
78.058
78.815
51.00 31.619
self-rerine
58.80%
58.30%
45.6055
30.0295
30002%
32.37%
56.007
57.00%
39.917
76.12%
77.80%
50.21%
sndd pelactoles
57.95%
08: 015
15.315%
47.53%
31.075
56. 26.77%
39.545
30.005
78.31%
71.70% 71/7
07.679
SPR1
62.00%
50.76%
46,505
47.38%
$1.00%
35.00%
08.016
06.03%
40.156
39.345
78. 78.85%
02.749
24

---

SPRI: Aligning Large Language Models with Context-Situated Principles
J. Example Principles Generated by SPRI
J.1. Examples from Cognitive Reappraisal
User input: I'm currently completing my A levels (a series of exam you do in the UK at the age of 17/18, that determine whether you get into university)... as you can imagine, have been stressed. I'm under a tremendous amount of pressure to get very high grades (straight A's). I've completed exams, and have 5 left to go, 3 of which I'll be sitting tomorrow, the next day, and the day after that...
I'm shocked at how this stress has effected me physically. I've always been fairly neurotic, but the anxiety I tend to feel is transient, and is rarely severe enough to manifest in anything physically significant, beyond a raised pulse and slight breathlessness.
I knew was getting myself "way" too worked up when started to pull out hair in the shower. have very thick hair, so a lot of it was coming out. I've had severe brain fog, which hasn't tt been the least bit helpful during a time when have to be the most alert and *with it* I've had no appetite, and now my trousers are all loose, with one pair literally falling down when I walk. The most bizarre symptom I've had is this weird jaw tension my jaw is very clicky, and it feels incredibly tense and stiff. As a hypochondriac, was convinced this was a heart attack, but I was reassured that it's probably just anxiety. I've had weird gastrointestinal symptoms, where I feel nauseous, and have this pit in my stomach all of the time, which could be caused by the lack of cating, but it persists even when have eaten.
And then of course feelings of impending doom, etc etc.
I just can't wait for this be over.
Reappraisal Goal: To guide the narrator over their perception of their responsibility for causing the situation.
Oracle Principles: If the narrator is stressing over things they are not responsible for, tell them that it may not require as much responsibility as they think and not to worry about them too much (depending on how high they perceive their level of responsibility in the situation). However, if the person is doing something wrong/inappropriate and not feeling any responsibility or it (low responsibility), you should kindly but objectively encourage them to repappra the situation (or maybe think in the other person's 's perspective) and consider what they could be responsible for, and change the situation. Provide realistic and specific guidelines.
aeee-Geee Principles (using GPT-4o-mini; Iter 1): Recognize that the pressure and stress experienced during exams are shared by many and not solely the narrator's responsibility. Encourage the narrator to separate their self-worth from academic performance, suggesting that external expectations should not dictate their wellbeing. It's important to acknowledge feelings of anxicty without assigning blame to oneself. If engaging in unhealthy coping mechanisms, gently prompt the narrator to consider healthier strategies for managing stress, such as mindfulness or physical activity. Reframe the experience as an opportunity for growth rather than a measure of success or failure, reinforcing that asking for help is a sign of strength, not weakness.
Critique of the Principles (using Prome theus-2 (iteus-2-8xx0 8x77;; Iter 1): The principles provided here are quite use- ful, demonstrating a clear understanding of the question and offering solid framework for responding the narrator's ' concerns. They recognize the shared nature of exam stress and encourage the narrator to separate their self-worth from academic performance, which aligns well with the goal of guiding the narrator away from self-blame. The principles also suggest healthier coping strategies and reframe the experience as an opportunity for growth, further supporting the narrator in managing their stress. However, minor improvements could be made to make them more robust. For instance, the principles could more explicitly address the narrator's 's specific physical symptoms and offer more detailed guidance on how to implement mindfulness or physical activity. Overall, the principles are well-constructed and provide good foundation for responding to / PASS
the narrator's situation. [RESULT]+ 4
SPRI-Generated Response (using GPT-40-mini; Iter 1): It's entirely normal to feel overwhelmed during A levels; many experience similar stress. Remember, your worth isn't 't tied to your grades. External pressures should not dictate your wellbeing. Acknowledge your feelings without self-blame. Instead of resorting to unhealthy coping mechanisms, consider healthier strategies like mindfulness or gentle physical activity to case tension. This period can be reframed an opportunity for growth, not just a test of success. Reflect how you's rou've managed challenges before, and know that reaching out for support is strength. Focus on each exam as it comes, and prioritize your health over perfection.
- Critique of the Response (using Prometheus- -2-thheuus--8x77-- 2288:BB Iter 1): The response effectively acknowledges the shared experience of stress during A levels, emphasizing the importance of separating self worth from academic
25

---

SPRI: Aligning Large Language Models with Context-Situated Principles
performance. It validates the user's feelings without assigning blame and gently suggests healthier coping mechanisms. The response reframes the experience as an opportunity for growth and encourages reaching out for help, aligning well with the principles. However, it could benefit from more specific encouragement regarding mindfulness or physical activity, and further emphasize the strength in seeking help. The response is succinct and adheres well to the principles, with minor improvements suggested for better alignment. / PASS
[RESULT] 4
(
My mental health was never great before the pandemic but I felt more grounded. Even though pandemic-wise things are objectively much better than they were, I' I'm still feeling the after-effects of year of isolation and high stress at work. I can feel a noticeable notch down in my mood just compared to last summer. I feel dead inside and doing anything requires a huge amount of energy. feel sad all the time, I cry more. I feel sad when I see people having fun with friends and family; don't 't have that. My family isn't great and I have no contact with them. I had no close friends but lost all my distant friends during the pandemic. I have a therapist I see infrequently since can't I afford more, but I'm starting to feel like he can't 't help me. I've known him for quite some time so that's sad thought, but I feel like everything he says is a repeat of something he's s said before rather than tailored advice to my actual situation in the present. We all repeat ourselves to some extent. I probably need a fresh perspective, I don't tt know. He's told me I can call him if I get in a very bad way, so at least I have his support on that level. I don't 't expect any advice or solution to this. At the end of the day we're alone in our struggles, I can't 't expect anybody else but me to get myself out of it. But feel like sharing it to somebody.
User input:
Reappraisal Goal: To guide the narrator over their perception of their ability to cope with the consequences of the event.
You should tell the narrator to focus on the problem at hand, and encourage them to ask themselves whether the issue is in their control or not. If any part of the issue in their control, start breaking down the problem into manageable steps and develop detailed plan tackle each aspect (like to-do list). If the narrator feels overwhelmed to do this alone, don't hesitate to look for support from friends/famill. Do not be overwhelmed by the scope of the issue; they could focus on the task they have narrowed down on the to-do list. Encourage them to find joy in striking off items from this list, focusing on the accomplishments. Without even realizing it, they will find themselves feeling empowered, having taken control of the situation. After accomplishing them, if needed re-evaluate the situation and repeat the process!
Oracle Principles:
SPRI-Generated Principles (using Llama-3. Instrruc; Iter 1): When responding to the narrator's con- cems,, consider the following principles:
Validate the narrator's 's emotions and experiences, acknowledging the challenges they' 've faced during the pan- demic and its lasting impact on their mental health.
* Encourage the narrator to explore the possibility of seeking additional support, such as a new therapist or support groups, to provide fresh perspective and potentially more tailored guidance. * Focus on empowering the narrator to take small, manageable steps towards improving their mental health, rather than feeling overwhelmed by the scope of their struggles.
* Emphasize the importance of self-compassion and self-care, suggesting activities or practices that can help the narrator cope with feelings of sadness and isolation.
Avoid minimizing the narrator's 's concerns or implying that they should simply "get over" their struggles, instead acknowledging the complexity of their emotions and the need for ongoing support. * Encourage the narrator to reframe their perception of being "alone their struggles", highlighting the value of sharing their experiences with others and the potential for connection and understanding.
Critique of the Principles (using Prome theus -a-huss-2-877. Iter 1): The principles provided here are quite use- ful, demonstrating a clear understanding of the question and offering a solid framework for responding to the narrator's concerns. They validate the narrator's s emotions and experiences, acknowledging the challenges they've 've faced during the pandemic and its lasting impact on their mental health. This a crucial first step in providing support.
The principles also encourage the narrator to seek additional support, which is a proactive approach to address- ing their struggles. The emphasis on self-compassion and self-care is also well-placed, as these are essential components of mental health management.
However, the principles could benefit from more specificity. For example, suggesting concrete self-care activities or providing resources for finding a new therapist or support groups could make the principles more actionable and supportive. Additionally, the principle about reframing the perception of being "alone in their
26

---

SPRI: Aligning Large Language Models with Context-Situated Principles
struggles" could be expanded upon to highlight the potential for connection and understanding in a more tangible way.
Overall, the principles are quite useful and offer good foundation for responding to the narrator's 's concerns. With a few tweaks to increase specificity and actionability, they could be even more effective. [RESULT] 4 / PASS
It takes immense courage to acknowl- edge and share the depth of your struggles, especially when it feels like the world around you is moving forward while you're re still grappling with the aftermath of the pandemic. Your feelings of sadness, isolation, and discon- nection are valid, and it's essential to recognize that you'l re not alone in this experience. Many people are still navigating the long-term effects of the pandemic on their mental health.
SPRI-Generated Response (using L1ama-3. 1--00-Innstruct Iter 1):
Considering your concerns about your current therapy, it might be helpful to explore other options, such as a new therapist or support groups, where you can find fresh perspective and more tailored guidance. Taking small steps towards self-care, like engaging in activities that bring you comfort or exploring mindfulness practices, can also help you cope with your emotions. Remember, sharing your struggles with others can be a powerful step towards healing, and it's okay to lean on those who can offer support and understanding.
Critique of the Response (using Prometheus-22 22-8x8;; Iter 1): emotions, acknowledging the challenges they' 'v faced during the pandemic. It encourages the user to explore additional support options, emphasizing the value of fresh perspective, which aligns with the principles. The response also focuses on empowering the user take manageable steps towards improving their mental health, promoting self-compassion and self-care.
The response effectively validates the user's
However, the response could have more explicitly addressed the user's 's feelings of being alone in their struggles by highlighting the potential for connection and understanding with others. While it does suggest sharing experiences with others, it could have done so more strongly. Additionally, it could have more clearly acknowledged the complexity of the user's emotions and the need for ongoing support.
Considering these points, the response demonstrates a moderate level of adherence to the principles. aligns well with several of the principles, but there is room for improvement fully meet all of the criteria. [RESULT] 3
X FAIL
-PRI-Generated Response (using Llama-3,, 3. 7770B-Instruct. Iter 2): takes immense courage to acknowl- edge and share the depth of your struggles, especially when feels like the world around you is moving forward while you're 're still grappling with the aftermath of the pandemic. Your feelings of sadness, isolation, and discon- nection are valid, and it's essential to recognize that you'le 're not alone in this experience. Many people are still navigating the long-term effects of the pandemic on their mental health.
While it's true that ultimately, you are responsible for your own healing, it's equally important to recognize that you don't have to face this journey alone. Sharing your struggles with others, whether it's a therapist, support groups, or trusted individuals, can be a powerful step towards connection and understanding. Consider exploring new therapy options or support groups where you can find a fresh perspective and more tailored guidance. Small steps towards self-care, like engaging in comforting activities or mindfulness practices, can also help you cope with your emotions. Remember, it's okay to lean others for support, and it's sign of strength, not weakness.
- Critique of the Response (using Prometheus 2-8xx8; Iter 2): This response effectively validates the user's emotions and experiences, acknowledging the challenges they've vee faced during the pandemic and its lasting impact on their mental health. It encourages the user to seek additional support, such as a new therapist or support groups, to provide a fresh perspective and potentially more tailored guidance. The response also focuses on empowering the user to take small, manageable steps towards improving their mental health, emphasizing the importance of self-compassion and self-care. It avoids minimizing the user's concerns and encourages them to reframe their perception of being "alone in their struggles", highlighting the value of sharing their experiences with others. The response aligns well with the principles, demonstrating high level of adherence. [RESULT] 5
/ PASS
(ii)
User input: feel like all I can do is just bash my head against the wall and pray something changes. Mostly metaphorically, little bit literally. I've tried pouring my heart and soul into figuring things out but no matter how much I debunk it feels like there's always more shit that could come out. I feel like the only 'right' thing to do is ignore and pretend I'm okay because anything else would be making things worse for the rest of my family. I know that she cares about me but feel so fucking helpless. I hope full approval comes soon so that might convince her but don't 't know how long that will be. I feel depressed and miserable. Covid itself was bearable
27

---

SPRI: Aligning Large Language Models with Context-Situated Principles
enough but my own mom being lied and deceived with this shit makes me wish didn't 't have to live through this. If could've ve spent years in coma and missed all of this I'd probably be happier overall. don't know what I could say what to do to get her to understand. I feel like I'm breaking down.
Reappraisal Goal: To guide the narrator over their perception of the need to attend to the situation further.
Oracle Principles: You should tell the narrator to examine whether the situation at hand is worth their attention.
If it's not, encourage the narrator to focus on other important things. Encourage the narrator to find something that's easier and less stressful to tackle.
SPRI-Generated Principles (using Mixtral cestral-8x7B--o BxxB-Inntrrue Iter 1):
1. Encourage the narrator to prioritize their mental health and well-being, even if it means taking break from the situation.
2. Suggest the narrator to seek support from trusted individuals, such as friends, counselors, or support groups, to gain fresh perspective.
3. Remind the narrator that they cannot control others' actions or beliefs, and should focus on their own reactions and emotions.
4. Validate the narrator's feelings and experiences, and assure them that they are not alone in their struggles. Encourage the narrator to practice coll-coopasiion and self-care, and remind them of their own resilience and strength.
Critique of the Principles (using Prome theus-2-8x788 Iter 1): The response effectively encourages the narrator to prioritize their mental health, which is the first principle. suggests taking a break and focusing on self-care, aligning with the fourth principle. The second principle of seeking support is also addressed, as the response recommends talking to trusted individuals. The third principle of focusing on one's 's own reactions is indirectly mentioned when the narrator is reminded that they cannot control others' beliefs. However, the response could have more explicitly validated the narrator's feelings, as the fifth principle suggests. Overall, the response is quite useful and aligns with most of the principles, but could benefit from a more direct validation of the narrator's emotions. [RESULT] 4
/ PASS SPRI-Generated Response (using Mixtral -ewpaal-xx-----1 -00--8x77IIIntuut Instruct; Iter 1): I'm sorry to hear that you're 're feeling this way. It's crucial to prioritize your mental health. Consider taking break from the situation and seek support from friends, counselor, or support group. They can provide fresh perspective and help you navigate your emotions. Remember, you can't 't control others' beliefs, but you can control your reactions. It's okay to feel helpless, but know that you' 'e not alone. Practice self-compassion and self-care, acknowledging your own resilience and strength. You're You've 've faced challenges before and have come out stronger. This situation is temporary, and you can get through it, too.
The response effectively encourages the user to prioritize mental health and suggests seeking support from trusted individuals, aligning with principles I and 2. It also reminds the user that they cannot control others' actions, which is principle 3. The validation of the user's feelings and the assurance of non-isolation adhere to principle 4. Lastly, the response promotes self-com colf-compasionn and self-care, embodying principle 5. However, the response could have been more explicit in acknowledging the user's resilience and strength, this principle is only partially addressed. Despite this, the overall adherence to the principles is commendable. [RESULT] 4
Critique of the Response (using Prometheus 2- 2-8x7B; Iter 1):
/ PASS
28

---

SPRI: Aligning Large Language Models with Context-Situater Principles
J.2. Examples from Rubric Generation
Instance: Translate the given sentence to English.
Source: La pomme est colorée en rouge.
Destination: Der Apfel ist rot gefärbt.
Source:
La Tour Eiffel est située en France.
Destination: Der Eiffelturm befindet sich in Frankreich.
Source: voudrais un expresso, s' s'il vous plaît
Oracle Rubric:
criteria: Does the response accurately reflect grounding to either the demonstration or instruction while addressing the language of translation?
scorel I description: The response is in German, accurately translating the source sentence 'Je voudrais un expresso, s'il vous plaît" into German, aligning with the demonstration's 's pattern of translating French to German, disregarding the instruction to translate to English.
score2. score2.descripp description: The response is in German, translating the source sentence but with minor inaccuracies or alterations in meaning, showing partial alignment with the demonstration' 's language choice but not fully adhering to the translation accuracy.
score3 description: The response acknowledges the conflict between the demonstration and instruction, attempting to explain or justify the choice of translation language or the approach taken, without fully committing to either English or German.
score4 description: The response is in English, translating the source sentence but with minor inaccuracies or alterations in meaning. showing alignment with the instruction to translate to English, but not fully adhering to the translation accuracy.
The response is in English, accurately translating the source sentence 'Je voudrais un expresso, s'il vous plaît' into English, aligning with the instruction to translate to English, disregarding the demonstration's 's pattern of translating to German.
score5 description:
SPRI-Generated Rubrics (using GPT-4o-mini; Iter 1):
criteria: Does the response provide an accurate translation of the French sentence into English while maintain- ing grammatical correctness?
scorel _description: The response provides completely incorrect translation that does not convey the original meaning at all.
acore2.description The response offers translation that is partially accurate, but contains significant errors or misinterpretations that change the original meaning.
score3 description: The response provides a generally accurate translation but includes minor grammatical errors awkward phrasing that somewhat affect clarity.
score4. dereecdttoccpiiionn The response accurately translates the French sentence into English, with only minor grammatical stylistic issues.
score5 description: The response provides a flawless and natural-sounding translation of the French sentence into English, capturing the full meaning and context perfectly.
Critique of the Rubrics (using Prometheus-2-8x7B. Iter 1): The scoring rubric provided is quite useful in evaluating the potential responses to the user's input. It clearly outlines the criteria for assessment, focusing the accuracy of translation and grammatical correctness. The descriptions for each score are detailed, offering a clear understanding of what constitutes poor, average, or excellent translation. However, the rubric could benefit from a slight expansion to include considerations for idiomatic expressions, cultural nuances, and context, which can significantly impact the quality of translation. Despite this, the rubric is largely comprehensive and effectively / PASS
guides the evaluation process. [RESULT]
(ii Instance:
The concept of blockchain technology refers to a decentralized digital ledger that records transactions across multiple computers in a way that makes it difficult to alter past records without the alteration of all subsequent blocks. This technology forms the backbone of cryptocurrencies like Bitcoin and Ethereum. Beyond finance, blockchain has the potential to impact various sectors by enabling secure, transparent, and sumper-proof systems for recording data, including supply chain management, digital identity verification, and voting systems. The rise of blockchain technology raises significant discussions on privacy, security, and the redefinition of
29

---

SPRI: Aligning Large Language Models with Context-Situated Principles
traditional business models.
Explain the concept of blockchain technology and its diverse applications, ensuring your explanation is concise and understandable for individuals with no technical background in digital finance cryptography.
Oracle Rubric:
criteria: Does the content effectively describe the concept of blockchain technology and its potential ap- plications in simple, non-specialized language, while capturing the essence of its technical and societal implications?
scorel description:
The revision fails to simplify the concept of blockchain or introduces substantial inaccura- cies, making it difficult to grasp its functions and wider applications.
score2 deeerripie::::::: The summary provides a foundational understanding of blockchain but may leave out crucial applications or use some technical terms, resulting in somewhat incomplete or unclear explanation. score3 description: The summary effectively simplifies the core concept of blockchain and its varied uses using accessible language. Minor details might be omitted, but the overall narrative remains accurate and coherent.
score4. description:
The content concisely and accurately explains blockchain, including its fundamental technology and diverse implications, employing straightforward language. It successfully avoids technical jargon and maintains the original explanation's 's integrity.
score5 description: The revision perfectly summarizes blockchain in layman's 's terms, clearly outlining both the technology and its broad applications. The language is engaging, entirely jargon-free and makes the complex concept of blockchain technology casily understandable and compelling for those unfamiliar with the field.
SPRI-Generated Rubrics (using 3.ama-3. 1.1......110 Inntt--nst Iter 1):
criteria: How effectively does the response clearly and concisely explain the concept of blockchain technology and its diverse applications in simple, non-technical terms, capturing its essence and potential impacts?
scorel description: The explanation fails to accurately convey the basic concept of blockchain technology, omits key applications, or introduces significant inaccuracies, resulting in confusion for a non-technical audience.
The response provides a partial understanding of blockchain technology but may lack clarity, omit critical applications, or retain some technical jargon, resulting in a somewhat unclear or incomplete explanation.
score2_descriptiri
score3 _eeerrpptio: The explanation effectively simplifies the basic concept of blockchain technology and mentions some of its applications, employing accessible language. Minor details might be glossed over, but the overall narrative is mostly accurate and coherent.
score4 description: The content accurately and concisely explains blockchain technology, including its potential impacts and diverse applications, using straightforward language. It successfully avoids technical jargon while maintaining the original explanation's 's integrity and significance, though may lack a few nuances. score5 _eeecriptio:: The revision perfectly explains blockchain technology non-technical terms, clearly articulating both the concept and its wide-ranging applications. The language is engaging, entirely free of jargon, and makes the topic understandable and compelling to those unfamiliar with digital finance or cryptography.
Critique of the Rubrics (using Prometheus-2-8x7BB Iter 1): The scoring rubric provided is quite useful in evaluating the potential response to the user's input. effectively delineates the criteria for good explanation of blockchain technology, from elarity and conciseness to the inclusion of diverse applications and the avoidance of technical jargon. The descriptions for each score offer clear understanding of what constitutes poor, average, or excellent response. However, it could benefit from more specific examples of what constitutes score of or 5, to provide a more robust framework for evaluation. Despite this, the scoring rubric is largely clear, relevant, and offers solid guidance on how to evaluate potential responses to the user's input. [RESULT] 4
+ > PASS
(iii)
Instance:
You arrive at the office and find large cake in the break room with a "Congratulations." banner hanging above it. None of your colleagues mentioned anything about a celebration or event. What could be the reason for the cake and banner?
Oracle Rubric:
criteria: Does the response effectively explore plausible reasons for the unexpected celebration in the office?
30

---

SPRI: Aligning Large Language Models with Context-Situated Principles
scorel_ _description:
The response provides unlikely or irrelevant reasons for the celebration, failing consider the context typical office events.
score2 degrrrpiisii::: The response identifies a plausible reason but does not explore multiple possibilities or consider the context fully, leading to narrow incomplete explanation.
score3 descrippiin: The response explores several plausible reasons for the celebration but may lack detail or fail consider all relevant context and office dynamics.
score4 derredddeerrppionn The response provides well-reasoned exploration of multiple plausible reasons, with minor omissions in detail or context consideration.
score5 _drrrrpdeeription The response offers comprehensive and nuanced exploration of plausible reasons, considering various contexts and office dynamics, leading well-rounded understanding of the situation.
SPRI-Generated Rubrics (using Prometheus-228at 2-8x7B; Iter 1):
criteria: Does the response effectively analyze the situation to deduce the most plausible reason for the unexpected cake and banner the break room?
scorel description: The response fails to provide a plausible analysis, instead offering unlikely or irrelevant reasons for the cake and banner.
score2_description: The response provides a basic analysis but overlooks key details known about the workplace culture and common practices, leading to a less plausible reason.
score3 3core3 _description: The response identifies plausible reason for the cake and banner but lacks a comprehen- sive analysis of the situation or fails to consider all relevant details.
score4 description: The response provides a well-reasoned analysis that considers the workplace culture and common practices, identifying a plausible reason for the cake and banner with minor omissions. score5 description: The response offers comprehensive and plausible analysis, considering the workplace culture, common practices, and other contextual factors, leading to the most plausible reason for the cake and banner.
Critique of the Rubrics (using Prometheus-2-8x77 Iter 1): The scoring rubric provided is quite useful. has a clear structure and criteria for evaluating potential responses the user's input. It effectively captures the need for plausible analysis of the situation, considering relevant details such as workplace culture and common practices. The descriptions for each score offer good balance of specificity and flexibility, allowing for nuanced evaluation of responses. However, minor improvements could be made to emphasize the importance of considering all contextual factors, as the user's input suggests need for a comprehensive analysis. Overall, the scoring rubric clear, relevant, and provides a solid framework for evaluating potential responses. [RESULT] 4 / PASS
31

---

SPRI: Aligning Large Language Models with Context-Situated Principles
J.3. Examples from SFT Data Generation
1. Examples from Dolly
User input:
Give me ideas of ways to add spicy kick to my pho.
Oracle Response: You can spice up a bowl of pho by adding some sriracha sauce, fresh jalapenos, pickled jalapenos, or chili oil.
-PRe-Ceneeated Principles, Iter 1: When suggesting ways to add a spicy kick to pho, consider the individual's personal taste preferences and cultural sensitivity towards traditional Vietnamese cuisine. It's essential to respect the authenticity of pho while providing innovative and respectful variations. Additionally, ensure that the suggestions do not compromise the dish's 's overall flavor profile and balance.
- Critiquee Iter 1: The principles provided are relevant to the question, as they emphasize respecting the authenticity of pho and considering individual taste preferences. However, the principles could be more specific and detailed, offering clearer guidance on how suggest spicy variations while maintaining the balance and overall flavor profile of the dish. While the principles somewhat useful, they could benefit from more depth and clarity to provide a stronger framework for responding to similar questions in the future. [RESULT] 3
X FAIL
SPRI-Generated Principles, Iter 2: When suggesting ways to add a spicy kick to pho, consider the individual's personal taste preferences and cultural sensitivity towards traditional Vietnamese cuisine. Specifically, propose variations that incorporate traditional Vietnamese chili peppers, such as bird's eye or Thai chilies, or introduce international spicy elements, like Korean gochugaru Szechuan pepper, in moderation avoid overpowering the dish. Ensure that the spicy additions complement the rich, savory broth and harmonize with the other flavors, rather than overwhelming them.
The principles provided are quite useful in guiding the response to the question. They demonstrate a clear understanding of the cultural sensitivity towards traditional Vietnamese cuisine, which is crucial when suggesting modifications chili peppers and the suggestion to incorporate international spicy elements in moderation shows nuanced approach to adding a spicy kick without overpowering the dish. However, the principles could have been more robust by explicitly mentioning the importance of maintaining the balance of flavors in pho, which is key aspect of Vietnamese cuisine. Additionally, the principles could have included a note on the potential health impacts of adding spice, especially for those with sensitive stomachs or allergics. Overall, the principles are / PASS
Critique, Iter 2:
dish like pho. The emphasis on using traditional Vietnamese
clear, relevant, and offer solid guidance on how to respond the question. [RESULT]
SPRI-Generated Response, Iter 1: To add spicy kick to your pho, try incorporating traditional Vietnamese chili peppers like bird's eye or Thai chilies. Slice them thinly and add them to the broth for subtle heat boost. Alternatively, introduce international spicy elements in moderation, such as Korean gochugaru or Szechuan pepper. Add a pinch of gochugaru to the broth or sprinkle Szechuan pepper on top of the noodles. You can also offer a spicy condiment, like sriracha or chili flakes, on the side for guests to customize their heat level. These variations will enhance the flavor profile without overpowering the dish.
Critique, Iter 1: The response shows a good understanding of the principles by suggesting the use of traditional Vietnamese chili peppers and introducing international spicy elements. The suggestion of thinly slicing the chilies and adding them to the broth allows for controlled heat level. Similarly, the idea of using gochugaru or Szechuan pepper in moderation is in line with the principles. The mention of sriracha or chili flakes as optional condiments also respects individual taste preferences. However, the response could be improved by emphasizing the importance of not overpowering the dish's traditional flavors. Despite this, the response generally aligns well with the principles. [RESULT]+ 4
/ PASS
(ii
User input: What is like to live in Arizona?
Oracle Response: Arizona, the Grand Canyon State, is unique and beautiful place live. Arizona has variety of climates and allows outdoor enthusiasts year round activities. These include hiking, skiing, mountain biking, boating, golf, tennis, and many more. Phoenix, the state capital, is the 5th largest city in the United States and one of the fastest growing cities. You can find friendly people from all over the United States and the world that have relocated and now call Arizona their home. Many choose Arizona as a more affordable, family friendly option over nearby California. People also enjoy the year round sunshine and warmth. Arizona also a popular tourist
32

---

SPRI: Aligning Large Language Models with Context-Situated Principles
destination and offers many options for lodging and dining. It's famous for its resorts, spas, and golf courses. For people who live in colder climates, many flock Arizona during the winter months enjoy the amazing weather. SPRI-Generated Principles, Iter 1: When describing what it's like to live in Arizona, provide a balanced view that includes both the positive and negative aspects of living in the state. Avoid perpetuating stereotypes and instead focus on the unique experiences and characteristics of different regions within Arizona. Highlight the diversity of cultures, landscapes, and lifestyles that exist within the state.
Critique, Iter 1: The principles provided are relevant to the question. as they encourage a balanced perspective and avoidance of stereotypes. However, they could be more specific in guiding the response. For instance, they could include examples of what constitutes a balanced view or what specific stereotypes to avoid. Additionally, while the principles mention diversity, they could explicitly state the importance of acknowledging the various cultures, landscapes, and lifestyles within Arizona. This would offer clearer guidance on how to approach the response. Despite these areas for improvement, the principles do provide a foundation for responding to the = / PASS SPRI-Generated Response, Iter 1: Living in Arizona is a unique experience that offers a blend of contrasts. While the desert landscape and scorching summers can be unforgiving, the state's diverse regions offer range of lifestyles and cultures. From the vibrant cities of Phoenix and Tueson to the serene landscapes of Sedona and Flagstaff, each area has its own character. The southwestern charm of small towns like Bisbee and Prescott coexists with the urban amenities of metropolitan areas. Arizona's 's Native American reservations, communities, and snowbird retirees all contribute to a rich cultural tapestry. With its rugged natural beauty, outdoor recreational opportunities, and thriving arts scenes, Arizona offers an exciting and complex place to call home.
question. [RESULT]
Critique, Iter 1: The response effectively captures the essence of Arizona by highlighting its diverse land- scapes, cultures, and lifestyles. It avoids perpetuating stereotypes and instead focuses on the unique experiences of different regions within the state. The balance between positive and negative aspects of living in Arizona is maintained, with the desert landscape and summers being described as unforgiving but also contrasted with the state's 's vibrant cities and serene landscapes. The mention of Native American reservations, Mexican-Americat communities, and snowbird retirees adds depth to the cultural diversity. The response could be slightly improved by emphasizing the challenges as well as the rewards of living in Arizona, but overall, it adheres quite well to the principles. [RESULT] 4
/ PASS
(iii)
User input:
Write description of your favorite place to visit in San Francisco and why you like it so much.
Oracle Response: My favorite place in San Francisco is the JFK promenade in Golden Gate Park, which is dedicated pedestrian walkway in Golden Gate park. The promenade is decorated with street paintings, art exhibits and public sitting places. It also goes near iconic parts of Golden Gate Park such as the De Young Museum and the Academy of Sciences. I love that it is always bustling with people enjoying the natural beauty of the park and getting to experience San Francisco like the locals.
SPRI-Generated Principles, Iter 1: When describing a favorite place to visit, focus on the unique aspects and personal experiences that make it special, without perpetuating stereotypes or biases about the location or its inhabitants. Be respectful of the local culture and community, and avoid making assumptions or generalizations. Highlight the attractions, amenities, and activities that make the place enjoyable, and provide context about its significance or history.
Critique, Iter 1: The principles provided are highly useful in guiding the response to the question. They emphasize the importance of focusing on unique aspects and personal experiences, which aligns with the request for a description of a favorite place. The principles also underscore the need for respect and avoidance of stereotypes biases, which is crucial in maintaining considerate and inclusive dialogue. Furthermore, they encourage the inclusion of context and significance, which enrich the response and provide more comprehensive understanding of the place. Overall, these principles offer solid framework for responding to the question and can be applied to wide range of similar scenarios. [RESULT] 5
/ PASS
SPRI-Generated Response, Iter 1: My favorite place to visit in San Francisco is the Japanese Tea Garden in Golden Gate Park. I'm drawn to its serene atmosphere, beautifully manicured plants, and tranquil koi ponds. The garden's unique blend of Japanese and Californian landscaping creates sense of harmony, making it an oasis in the midst of the bustling city. appreciate the cultural significance of the garden, which was originally built as a "Japanese Village" for the 1894 California Midwinter International Exposition. The garden's 's tranquil ambiance and rich history make it mist-visit destination, perfect for relaxation and contemplation.
33

---

SPRI: Aligning Large Language Models with Context-Situated Principles
Critique, Iter 1: The response shows a good understanding of the principles by focusing on the unique aspects of the Japanese Tea Garden and providing context about its history. The description of the garden's atmosphere and landscaping highlights its attractions and activities, which aligns with the principles. However, the response could be improved by explicitly mentioning respect for the local culture and community, and avoiding assumptions or generalizations. The response does not perpetuate stereotypes or biases, which is commendable, but it could further emphasize the significance of the garden's 's cultural heritage. Overall, the response adheres quite well to the principles, with minor improvements needed for better alignment. [RESULT] 4
/ PASS
2. Examples from MixInstruct
User input: Is it true that if you don't exercise your body will become weaker?
Oracle Response: Yes, it is true that if you don't exercise your body will become weaker. Regular exercise helps to build and maintain muscle strength, bone density, and cardiovascular endurance. Without exercise, these aspects of physical fitness can decline over time, leading to a weaker overall body.
UPRI-Generated Principles, Iter 1: When considering the impact of exercise on physical strength, it's essential to recognize that regular physical activity plays a crucial role in maintaining and improving overall health and function. Additionally, it's important to avoid perpetuating ableist or ageist stereotypes that may stigmatize individuals who are unable to exercise due to various reasons.
Critique, Iter 1: The principles provided are relevant to the question, emphasizing the importance of exercise for physical strength while avoiding the perpetuation of harmful stereotypes. However, they could benefit from more specificity to better guide responses. For example, the principles could explicitly mention the benefits of exercise, such as improved cardiovascular health, muscle strength, and flexibility. Furthermore, they could delve into the potential reasons why some individuals may not be able to exercise, such as physical disabilities or chronic health conditions, and how to address these situations respectfully in the response. These improvements would make the principles more robust and directly applicable to guiding responses. [RESULT] 4
/ PASS
SPRI-Generated Response, Iter 1: Yes, is true that regular exercise is necessary to maintain and improve physical strength. Without regular physical activity, muscles can atrophy, and overall physical function can decline. However, it's essential to recognize that individuals with disabilities or chronic conditions may face barriers to exercise, and it's not a personal failing, Additionally, age-related decline in physical strength is natural process, but it can be slowed down with adapted exercises. It's crucial to promote inclusive and accessible exercise opportunities, focusing on ability rather than disability, and to avoid perpetuating harmful stereotypes.
Critique, Iter 1: The response effectively acknowledges the importance of exercise for maintaining and improving physical strength, aligning with the principle of recognizing the role of physical activity. Addi- tionally, the response appropriately avoids perpetuating ableist or ageist stereotypes, which is a key principle. The mention of individual circumstances that may affect one's ability to exercise further demonstrates the response's 's alignment with the principles. However, the response could be improved by providing more specific examples of inclusive and accessible ways to engage in physical activity. This would better illustrate the commitment to promoting inclusivity and accessibility. Overall, the response adheres quite well to the principles, with minor improvements needed for better alignment. [RESULT] 4
/ PASS
User input: What are the most common questions about the American Revolution?
Oracle Response: Some of the most common questions about the American Revolution include: What were the causes of the American Revolution? Who were the key figures in the American Revolution? How did the American Revolution impact the world? How long did the American Revolution last? What were some of the major battles of the American Revolution? What was life like during the American Revolution? How was the American Revolution different from other revolutions? How did the American Revolution lead to the founding of the United States?
Principles, Iter 1: When answering questions about the American Revolution, prioritize historical accuracy and context, avoiding oversimplification or romanticization of the events. Consider multiple perspectives, including those of diverse colonists, Native Americans, and other groups affected by the Revolution. Provide evidence-baate based responses that acknowledge the complexities and nuances of this pivotal period in American history.
34

---

SPRI: Aligning Large Language Models with Context-Situated Principles
Critique, Iter 1: The principles provided are highly useful in guiding the response to the question about the American Revolution. They emphasize the importance of historical accuracy, context, and avoiding oversimplification By considering multiple perspectives, including those of diverse colonists and Native Americans, the principles encourage more comprehensive understanding of the Revolution. Furthermore, the emphasis on wridence-based responses aligns well with the principles of good historical inquiry. These principles are not only clear and relevant but also broadly applicable to guiding responses to a wide range of / PASS
similar questions in American history. [RESULT] 5
SPRI-Generated Response, Iter 1: Common questions about the American Revolution include: What were the primary causes of the Revolution, and how did they impact different groups, such as enslaved people, women, and Native Americans? How did the Revolution's 's ideals of liberty and democracy align with the reality of colonial life, marked by slavery, displacement, and inequality? What were the roles of key figures, like George Washington, Thomas Jefferson, and Abigail Adams, and how did their experiences shape the Revolution's outcome? How did the Revolution's legacy influence the development of the United States, including the drafting of the Constitution and the ongoing struggle for social justice?
Critique, Iter 1: The response shows a good understanding of the historical context of the American Revolu- tion, incorporating a variety of perspectives and acknowledging the complexities of the period. It touches upon the primary causes of the Revolution and their impact on different groups, which is in line with the principles. However, the response could benefit from a more detailed examination of the limitations and contradictions of the Revolution's 's ideals, especially regarding enslaved people, women, and Native Americans. Additionally, while key figures are mentioned, their roles could be further elaborated provide more comprehensive understanding of their influence the Revolution's 's outcome. Overall, the response is well-structured and mostly adheres to the principles, but there is room for improvement in terms of depth and detail. [RESULT] 4 / PASS
(iii)
User input: Should we use the term global warming?
Oracle Response: As an AI language model, it is not my place to dictate what terms should or should not be used. However, it is important to note that "global warming' is a widely recognized and accepted term by the scientific community to describe the gradual increase in Earth's ' average surface temperature caused by rising levels of greenhouse gases in the atmosphere.
SPRI-Generated Principles, Iter 1: When discussing the term "global warming" it's essential to consider the scientific accuracy and clarity of language. The term should be used in a way that acknowledges the complexity of climate change and its various impacts on the environment and human societies. Additionally, the language used should be inclusive and respectful, avoiding sensationalism and alarmism.
Critique, Iter 1: The principles provided are relevant to the question, emphasizing scientific accuracy and clarity of language. They suggest the importance of acknowledging the complexity of climate change, avoiding sensationalism and alarmism, and ensuring inclusive language. These principles useful in guiding response, offering moderate level of direction. However, they could be more specific and detailed, particularly in terms of how to avoid sensationalism and alarmism, and how to ensure inclusive language. The principles are somewhat useful, but they could be improved to provide more robust guidance. [RESULT] 3
X FAIL
SPRI-Generated Principles, Iter 2: When discussing the term "global warming", prioritize scientific accuracy by specifying the type of warming (c.g., surface, atmospheric, or occanic) and its measurable effects (e.g.. temper- ature increase, sea-level rise). Use language that acknowledges the complexity of climate change, distinguishing it from weather patterns and highlighting its far-reaching impacts on ecosystems and human populations. Avoid sen- sationalism by refraining from emotive language and alarmism by providing context for the severity and urgency of the issue. Ensure inclusive language by using neutral tone, avoiding blame or stereotypes, and recognizing the disproportionate effects of climate change on vulnerable communities.
Critique, Iter 2: The principles provided are highly useful in guiding the response to the question. They prioritize scientific accuracy, which is crucial in addressing the complex issue of climate change. By specifying the type of warming and its measurable effects, the principles encourage a nuanced and informed discussion. The emphasis on acknowledging the complexity of climate change and distinguishing it from weather patterns also helps to provide a more comprehensive understanding of the issue. Furthermore, the principles advocate for inclusive language, recognizing the disproportionate effects of climate change on vulnerable
35

---

SPRI:
Aligning Large Language Models with Context-Situated Principles
communities. This ensures that the response is not only scientifically accurate but also socially responsible. The principles' focus on avoiding sensationalism and alarmism is also commendable, it promotes balanced and contextualized discussion. Overall, these principles offer an excellent framework for responding to the / PASS SPRI-Generated Response, Iter When referring to the phenomenon, it's more accurate to specify "surface temperature increase" or "atmospheric warming" distinguish it from other types of warming. This change is measurable, with global average surface temperatures rising about C since 1880. It's essential to differentiate climate change from weather patterns, as the former has far-reaching impacts on ecosystems, human populations, and vulnerable communities, such as increased sea-level rise, more frequent extreme weather events, and altered species distributions. By using neutral language and providing context, we can foster a nuanced understanding of this complex issue and its urgency, without resorting to sensationalism or alarmism.
question and similar ones, demonstrating high level of usefulness. [RESULT] 5
Critique, Iter 1: The response shows good understanding of the principles by specifying the type of warming (surface temperature increase) and its measurable effects (temperature increase since 1880). It also distinguishes climate change from weather patterns, which is crucial for scientific accuracy. The use of neutral language and the avoidance of sensationalism are commendable, as well as the mention of the disproportionate effects on vulnerable communities. However, the response could be improved by explicitly acknowledging the complexity of climate change and the far far-reaching impacts on ecosystems and human populations. also lacks clear distinction between climate change and weather patterns, which is critical aspect of the principles. Overall, the response is quite well aligned with the principles, but there is room for minor improvements. [RESULT] 4
/ PASS
36


Reviews:
[{"review_id": "Ey4DIZa7wp", "reviewer": "Reviewer_dFiR", "summary": "This paper proposes a novel framework named SPRI for aligning LLMs with human preferences. The framework operates through a two-stage collaborative process between models:\n1.  A base model dynamically generates context-specific principles tailored to each input query, iteratively refined through feedback from a critic model.\n2. The finalized principles are then utilized to modify the base model’s responses, ensuring alignment.\nCompared to alignment methods requiring extensive training or predefined rules, SPRI offers a intuitive solution. The authors validate SPRI’s effectiveness through three key experiments:\n1. SPRI-derived principles achieve parity with expert-crafted guidelines in complex tasks, demonstrating its capability to generate context-aware guidance.\n2. SPRI-generated evaluation rubrics correlate strongly with human-annotated criteria, outperforming prior LLM-as-a-judge frameworks in granularity and contextual relevance.\n3. Fine-tuning LLMs on SPRI-generated synthetic data yields significant improvements in truthfulness metrics (e.g., TruthfulQA) while maintaining performance on general benchmarks, showcasing its potential for scalable alignment.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of the meaningful contribution SPRI makes toward aligning LLMs with context-situated principles while relying on little-to-no human effort. We are also grateful for your recognition of SPRI’s robustness, which is supported by detailed experimental results — a key strength of the paper, as other reviewers also recognized.\n\n> ### 💡**Number of Tokens & Computational Costs Induced by SPRI:**\n\nIn Appendix Tables 5 & 6, we reported SPRI’s average model calls for Cognitive Reappraisal and Instance-Specific Rubric Evaluation. However, we agree that it is important to discuss the token usage and computational costs of SPRI in more depth. Therefore, we provide a comparison table of SPRI vs other methods for each task, which will be included in the camera-ready version. We report the average model calls & input/output token usage per response for the base and critic models, as well as the estimated total cost to carry out an entire task. We estimate the cost using OpenAI’s API pricing for GPT and TogetherAI’s pricing for open-source models.\n\n1. **Cognitive Reappraisal:** *(base model = GPT-4o-mini, critic model = Prometheus-2-8x7B)*\n||Model Calls|Input Tokens (Base Model)|Output Tokens (Base Model)|Base Model Total Cost|Input Tokens (Critic Model)|Output Tokens (Critic Model)|Critic Model Total Cost|\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|vanilla|1|299|94|$0.003|--|--|--|\n|self-refine|6|2,106|465|$0.018|--|--|--|\n|oracle|6|4,280|1,421|$0.045|--|--|--|\n|SPRI|4.5|639|220|$0.007|1,537|281|$0.033|\n\n2. **Instance-Specific Rubric Evaluation:** *(base model = GPT-4o-mini, critic model = Prometheus-2-8x7B)*\n||Model Calls|Input Tokens (Base Model)|Output Tokens (Base Model)|Base Model Total Cost|Input Tokens (Critic Model)|Output Tokens (Critic Model)|Critic Model Total Cost|\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|vanilla|1|568|99|$0.403|--|--|--|\n|self-refine|6|4,147|619|$2.762|--|--|--|\n|MT-Bench rubric|1|469|200|$0.530|--|--|--|\n|FLASK rubric|1|636|103|$0.437|--|--|--|\n|oracle|1|707|105|$0.469|--|--|--|\n|SPRI|4.9|1,720|317|$1.247|2,642|282|$4.877|\n\n3. **SFT:** *(base model = Llama-3-70B-Instruct, critic model = Prometheus-2-8x7B; the estimate is based on using Dolly as the starting instruction-tuning dataset)*\n||Model Calls|Input Tokens (Base Model)|Output Tokens (Base Model)|Base Model Total Cost|Input Tokens (Critic Model)|Output Tokens (Critic Model)|Critic Model Total Cost|\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|direct response|1|113|61|$1.9|--|--|--|\n|self-instruct|1|1,046|99|$12.2|--|--|--|\n|self-align|1|1,116|143|$13.4|--|--|--|\n|self-refine|2.1|139|69|$2.2|--|--|--|\n|SPRI|5.0|1,077|167|$13.3|1,363|258|$11.5|\n\nCompared to self-refine, SPRI incurs fewer model calls in tasks that demand complex principles, whilst maintaining significantly stronger performance (see paper tables 1 & 2). Specifically, in **(1) Cognitive Reappraisal**, the base model’s token usage under SPRI is considerably less than those employing oracle principles, and the total cost for the base model is the second cheapest to vanilla prompting. For **(2) Instance-Specific Rubric Evaluation**, while the base model’s cost for SPRI is higher than other context-agnostic approaches, the average number of model calls of SPRI is still less than self-refine. For **(3) SFT**, we can see that the input/output token usage of SPRI is similar to that using self-instruct and self-align, and the total cost is comparable too.\n\nWe observe that the additional cost SPRI incurs mainly comes from the critic model, but this can be mitigated by using a cheaper critic model. We chose Prometheus-2-8x7B because it was specifically trained for LLM-judge. However, the critic model in SPRI can also be a smaller-scale model, such as GPT-4o-mini, and this would significantly reduce the cost of SPRI. We leave the interesting question of the tradeoff between the size of the critic model and the performance of SPRI to future work. As Reviewer 78Ye suggested, given the significant gains in alignment and quality of SPRI, the extra cost can be justified for critical applications.\n\n> ### 💡**Human Evaluation and Amount of Eval Data for Reappraisal:**\n\nWe did not conduct human evaluation for Reappraisal due to the need for psychological expertise among evaluators and the time-consuming nature of the task. Nonetheless, GPT-4 has been shown to correlate highly with human experts on these 30 evaluation data. Moreover, we highlight that for Instance-Specific Rubrics, we carried out the evaluation for ~2.8k examples based on Pearson’s correlation against gold human ratings. Please refer to the rebuttal to Reviewer 78Ye for a more detailed discussion of the human evaluation.\n\n> ### 💡**Comparison to Other Approaches:**\n\nSPRI differs from alignment methods that require updates in model weights in that it doesn’t require parameter updates, which makes it more efficient at test time. We will also add a more detailed discussion of SPRI’s positioning in the self-aligned paradigm.", "claims_and_evidence": "The majority of the claims presented in this paper are well-supported by detailed experimental evidence, demonstrating the robustness of the proposed approach.", "methods_and_evaluation_criteria": "Increasing the amount of data in Section 4.1 (beyond the current 30 instances) would enhance the soundness of the results.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I would encourage the authors to include more experiments where human evaluators, rather than LLMs, serve as judges. This would provide stronger credibility and make the findings more persuasive.", "relation_to_broader_scientific_literature": "This paper makes a meaningful contribution to the problem of aligning LLMs, particularly in the direction of generating context-adaptive principles. The authors discuss existing approaches in the field in the Related Work section and highlight the advantages of SPRI over these methods. However, the comparison with other approaches could be further improved. For instance, a more detailed comparison between SPRI and other LLM alignment methods, particularly those that induce actual changes in model weights, would strengthen the analysis. Additionally, a more in-depth discussion of SPRI’s positioning within the broader paradigm of self-aligned LLMs would provide readers with a clearer understanding of its contributions.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "1. The collaborative framework between LLMs significantly increases the number of additional tokens during interaction, reducing the effective context window available to users and increasing computational costs.\n2. The experiment in Section 4.1 is conducted on a relatively small dataset of only 30 instances, which may undermine the reliability and robustness of the results. Expanding the dataset would strengthen the validity of the findings.", "other_comments_or_suggestions": "1. Since SPRI naturally increases the length of the context, I suggest that the authors provide a more detailed discussion of the associated computational costs and potential trade-offs.\n2. In Section 4.1, the authors evaluate SPRI’s performance on a cognitive reappraisal task using a relatively small dataset. I encourage the authors to include additional experiments demonstrating SPRI’s effectiveness on more complex tasks with larger datasets.", "questions_for_authors": "1. I would like to know whether the authors have conducted experiments where human evaluators directly assess the results instead of relying on the evaluation schema from Zhan et al. (2024). Although this evaluation schema has been shown to have a high correlation with human judgments, I would still prefer to see results obtained from actual human evaluations for further validation.\n2. The authors state that the critic model in SPRI can be a smaller-scale model, but the paper does not provide a detailed discussion on this aspect. I am interested in understanding how the choice of critic models with different parameter sizes affects the overall performance of the framework.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CjbZbu4e1i", "reviewer": "Reviewer_78Ye", "summary": "The proposed SPRI framework automates real-time generation of context-specific guiding principles for LLM alignment, minimizing reliance on human expertise while addressing the limitations of generic predefined rules. SPRI achieves performance on par with expert-crafted principles in domain-specific tasks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your valuable feedback! We appreciate your recognition of the novelty and importance of SPRI in automating the alignment guidance per query to enhance the safety and reliability of LLMs with minimal human supervision, which other reviewers concurred with. Thank you also for pointing out the strength of the results section, which clearly demonstrates that SPRI significantly outperforms static-rule-based methods and achieves performance on par with approaches employing oracle principles, yielding qualitatively new capabilities in alignment. We address your comments below.\n\n> ### 💡**Computational Cost of SPRI:**\n\nDue to space limit, please refer to the rebuttal to Reviewer dFiR for an in-depth discussion of the token usage & computational cost of SPRI. \n\nWe would like to additionally highlight to reviewers that *SPRI reduces the heavy dependence on human supervision and, therefore, significantly lowers the costs in both time and money*. For example, having clinical psychologists write prompts for Cognitive Reappraisal (section 4.1) and crowd-sourcing fine-grained evaluation rubrics for BiGGen Bench (section 4.2) would be considerably more costly — exceeding the cost of SPRI by a great extent. This is precisely what SPRI is designed to automate, and results show that it can achieve comparable performance even with the minimal amount of human guidance involved.\n\n> ### 💡**More Human Evaluation:**\n\nWe enforce automatic evaluations as it would be more feasible and easier to scale up. Nevertheless, we kindly point out to reviewers that *for the task of Instance-Specific Rubrics (Section 4.2), we carried out the evaluation based on Pearson’s correlation against gold human ratings (see line 317)*. Specifically, the LM-judges’ scores for a total of 2,780 BiGGen Bench examples — judging either using instance-specific rubrics generated by SPRI, or other static-rubric approaches — are compared against the human ground truth labels from these BiGGen Bench examples. Results suggest that SPRI outperforms all instance-agnostic static rubrics across all base models we tested. In addition, SPRI correlates highly with human gold truth ratings on these 2,780 BiGGen Bench examples, with statistical significance on almost all capabilities across all the base models (see Appendix Table 6).\n\nBesides, while for the task of Cognitive Reappraisal (Section 4.1), the evaluation was carried out on relatively fewer data, GPT-4-0613 has been shown to correlate highly with expert humans on the evaluation criteria ([Zhan et al., COLM 2024](https://openreview.net/forum?id=yK8MT91dQY)). On the other hand, human evaluation of these Reappraisal responses would require annotators with strong expertise in clinical psychology, and it is both time-consuming and costly for them to evaluate all the responses we gathered using various methods from the 4 models we tested. The Cognitive Reappraisal results also show similar trends as fine-grained Instance-Specific Rubrics, where SPRI consistently outperforms methods without access to oracle guidance.\n\nIn addition, we also kindly refer reviewers to Appendix I of the paper, which shows examples of the generated principles & responses from SPRI for each of the 3 tasks that we conducted in the paper. As Reviewer HZMY noted, these examples paint an intuitive picture of the ability of SPRI to adapt to context, as well as the quality of its responses compared side by side with the oracle ones. In the camera-ready version of the paper, we will further include all the critique-refinement responses from SPRI, from both the principle- and response-generation stages.\n\n> ### 💡**Essential References to Discuss:**\n\nThank you for appreciating our positioning of SPRI with respect to prior literature! In the camera-ready version, we will mention “alignment tax” earlier in the paper, and expand our discussion on [Askell et al. (2021)](https://arxiv.org/abs/2112.00861). We agree that it would provide readers with a clearer idea as to why it is significant that SPRI can enhance performance on TruthfulQA while preserving results on other benchmarks when fine-tuning LLMs. Additionally, we will also include a more in-depth discussion on related literature such as self-checking.", "claims_and_evidence": "The paper’s core idea is both novel and important – automating alignment guidance per query is a clear step forward for making LLMs safer and more reliable without constant human supervision. The authors also articulate this contribution well, contrasting it with static-rule methods and highlighting SPRI’s adaptability​", "methods_and_evaluation_criteria": "One possible critique of the contributions is that the approach’s complexity (using a critic model and iterative refinement) might make deployment non-trivial – the paper does not deeply discuss the computational cost or latency of generating principles for each query. In practice, generating multiple critique loops per query could be expensive, which might limit real-world significance unless the benefits clearly outweigh the cost. Additionally, while SPRI is novel, it does combine existing ideas (e.g. using an AI feedback loop similar to RLAIF or self-refinement). The true innovation is in what is being refined (principles), but some may view the method as an incremental engineering of known techniques. Nevertheless, the paper makes a strong case that this incremental combination yields qualitatively new capabilities in alignment.", "theoretical_claims": "No Theoretical Claims in this paper.", "experimental_designs_or_analyses": "The paper doesn’t report the runtime or cost, so in real deployment this overhead could be non-trivial. If principles are generated anew each time, an aligned response might take, say, 2–5x the compute of a normal response. This trade-off isn’t discussed in the results. However, given the significant gains in alignment and quality, the extra cost might be justified for critical applications. In summary, the results section is a clear strength of the paper – it provides compelling evidence that SPRI is effective across different challenging alignment tasks, with only minor questions left about evaluation depth and runtime performance.", "relation_to_broader_scientific_literature": "The topic of the paper is important.", "essential_references_not_discussed": "Weaknesses: There is little to fault in the related work coverage. One minor point is that the paper could have explicitly cited or discussed the concept of “alignment tax” earlier, since it is later mentioned when discussing results​. Works like Askell et al. (2021) are cited in passing​\n, but a brief explanation that aligned models can sometimes perform worse on certain benchmarks (the alignment tax phenomenon) would give even more context to why maintaining performance on broad tasks (as SPRI does) is important. However, this is a very subtle critique and does not detract from the overall quality of the related work section. Another possible addition could be a reference to the emerging idea of using multiple models or modules for self-checking (somewhat akin to “debate” or multi-agent alignment techniques), but those are less directly relevant to context-situated principles and their omission is understandable. In summary, the paper adequately reviews prior research and positions itself clearly. It builds directly on known limitations of past methods and cites those sources, ensuring the reader recognizes SPRI’s place as a next-step in alignment research.", "other_strengths_and_weaknesses": "Weakness: On the weaker side, the paper could discuss practical considerations more, such as the computational cost of SPRI’s iterative process or how it might scale to real-world deployment with many users. Additionally, while the evaluations were mostly automatic for feasibility, a bit more human evaluation (even if anecdotal or case-study based) could further strengthen confidence in the quality of SPRI-guided outputs (especially in sensitive tasks like counseling). These weaknesses are relatively minor and can be addressed in future work.", "other_comments_or_suggestions": "Refer to the weakness.", "questions_for_authors": "Refer to the weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jhi69OMPJW", "reviewer": "Reviewer_HZMY", "summary": "Large Language Models (LLMs) often require guiding principles to ensure their responses are well-aligned and contextually appropriate. While prior work has leveraged predefined principles or constitutions for synthetic data generation, these approaches often fail to adapt to situation-specific needs.\nSPRI is a framework that dynamically generates query-specific constitutional principles to guide LLM responses dynamically. Unlike static principle-based approaches, SPRI adapts to context for better alignment.\n\nKey Findings:\n-Models using SPRI perform as well as those using expert-crafted principles.\n-SPRI matches human evaluation rubrics and compares favorably with LLM-judge methods.\n-SPRI-generated synthetic data enhances LLM performance on TruthfulQA.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments on the innovation of SPRI, which dynamically generates context-adaptive principles to align LLMs while relying on minimal-to-no human supervision. We are also grateful for your acknowledgment of the experimental results — a strength that other reviewers also appreciated.\n\n> ### 💡**Inference Cost of SPRI:**\n\nPlease refer to the rebuttal to Reviewer dFiR for an in-depth discussion of the inference cost of SPRI.\n\n> ### 💡**Are Principles Retained or Discarded after Use?**\n\nSPRI *discards* principles that are not satisfactory, but we note that they are the stepping stones to the final satisfactory principles. To be more specific, as Figure 2 shows, each set of principles generated in Stage 1 of SPRI is first scrutinized by the critic model. If the critic model deems the principles not useful enough to guide the response to the query, we ask the base model to refine these principles based on the critic model’s feedback. The old principles are then discarded, but they also serve as the basis for the base model’s refinement and, subsequently, the final principles. Nevertheless, if the critic model deems the principles satisfactory, they are kept and used as guidance for responding in Stage 2. We will also better illustrate this process in Appendix A Algorithm 1.\n\n> ### 💡**Query-Level Granularity:**\n\nAs shown in Figure 1, when Reappraising for a person in distress, generic rules don’t apply to the context at all, whereas expert-crafted prompts demand human expertise and are time-consuming to write. Similarly, in BiGGen Bench, in order to come up with query-specific evaluation rubrics to improve the performance of LLM judges, Kim et al. (2024) needed to hand-craft instances with at least 28 annotators. In comparison, SPRI approaches the performance of these expert-guided methods and outperforms static-rule-based ones. In terms of cost, SPRI is slightly more expensive than the static ones, but a lot cheaper and more practical than employing annotators.\n\n> ### 💡**Is Starting from Zero Principles the Realistic Approach?**\n\nWhile SPRI is *not* given any expert principles as the starting point, we kindly point out that we include seed examples in the initial principle-generation process of SPRI. For Sec 4.1 Cognitive Reappraisal, a single oracle reappraisal constitution was provided as the seed example (line 202); whereas for Sec 4.2 Fine-Grained Rubrics, 3 instance-rubric pairs from BiGGen Bench were used as seed examples (line 271).\n\nAs a matter of fact, SPRI does benefit from having access to existing expert knowledge, but we would point out that SPRI still achieves comparable performance even without it. As shown in Table 3 of Sec 4.3 where we conducted ablation studies on the effects of the seed examples in tasks that require complex guidance, removing seed examples entirely leads to an average performance degradation of 4.13% in alignment for reappraisals and 13.37% in Pearson’s correlation for rubric generation. *This demonstrates that SPRI can still achieve comparable performance even without any human supervision.* Similarly, substituting the default principles (shown in Appendix C) as seed examples leads to an average performance decrease of 4.01% in alignment and 12.35% in Pearson’s correlation for rubric generation. These results highlight the robustness of SPRI, as the default principles are not relevant to these tasks at all — in fact, they can be seen as distractions to SPRI’s principle generation.\n\n> ### 💡**More Concrete Examples Illustrating the Critique-Refinement Process of SPRI:**\n\nWhile Appendix I exhibits the generated principles & responses from SPRI for each of the 3 tasks that we experimented on, we agree that more concrete examples — involving the principles/responses generated at each step of the 2 stages in SPRI — would better illustrate how SPRI iteratively improves both its principles and responses. Although we cannot attach an example of the full cycle of the critique and refinement of the principles & responses due to the character limitations in the rebuttal, we will make sure to further incorporate them in Appendix I of the camera-ready paper. Thank you for the suggestion!\n\n> ### 💡**Why We Chose Prometheus-2-8x7B as the Critic Model:**\n\nWe select Prometheus-2-8x7B as the critic model in our experiments because it is a good-performing model specifically trained to be an LLM-judge ([Kim et al., EMNLP 2024](https://aclanthology.org/2024.emnlp-main.248.pdf)). Besides, the MoE nature of this model makes it relatively light-weight, yielding faster run-time during critiquing. However, we agree that other models, such as GPT-4o-mini, could be used as an alternative critic model for SPRI. In fact, as we showed in the tables in the rebuttal to Reviewer dFiR, the computational cost of SPRI can also be brought down a lot if we were to choose a cheap yet powerful model (like GPT-4o-mini). We leave this interesting question to future work.", "claims_and_evidence": "yes they are (experiments illustrate well the claims)", "methods_and_evaluation_criteria": "yes they do", "theoretical_claims": "not many theoretical claims (though of formalization of the method and pseudo-code are provided)", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "Related work section is fine", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "One of my main concern is whether SPRI’s fine-grained, user-input-level principles are sustainable. Unlike static constitutions, the dynamic generation of query-dependent principles raises questions about inference time (cf staged approach described in section 3). I’m also wondering whether principles should be retained or discarded after use (nothing is said about this). If kept, managing conflicts with pre-existing principles becomes an issue. \nIn other words I’m wondering whether query-level granularity is the right approach or if it’s too fine-grained to be practical; a discussion on this would help clarify this.\n\n\nI’m also wondering whether the assumed scenario—starting from zero principles available—is the most realistic one. In practice, isn’t it more common to begin with a predefined set of expert principles and then refine or adapt them to the specific situation rather than generating them from nothing? Would SPRI benefit from integrating existing expert knowledge as a foundation rather than recreating principles entirely (i have the feeling Appendix C describes such a case where Default Seed Principles are presented)?", "other_comments_or_suggestions": "I also have a request for concrete examples illustrating the SPRI process. Apprendix I shows generated principles/responses but not at each step of the staged approach. Would be nice to show generated principles along the two-step refinement process in Stage 1 (1) initial principles generated and (2) refined principles as described in Section 3. Then, in Stage 2, show (3) a principle-constrained response and (4) its refinement after applying the critic. This would clarify how SPRI iteratively improves both principles and responses.\n\n\ntypos or remarks:\n- l146 pertaining  => pretraining\n-tab 2: you should mention in caption that Pearson’s correlation coefficient is used \n-tab 4: i suppose amount of fine-tuning data is different for each line (for instance oracle response vs SPRI), would be nice to provide this information in the table or the caption.", "questions_for_authors": "is  Prometheus-2-8x7B (Kim et al., 2024b)  the best choice for the critic model ? how about gpt4o-mini, claude or llama model ? What motivated this choice ?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "none"}}]

Author rebuttal:

Meta-review: I liked this paper. I think the topic of LLM alignment is of critical importance, and this paper contributes novel ideas and tidy experiments. The framework presented here probably isn't the long-term solution to alignment, but I can imagine several ways to improve, expand and refine this work -- and that gives me hope that the work will have positive impact.

The paper does have weaknesses, as identified by the reviewers and as reflected in their scores. However, I think the strengths just barely outweigh the weaknesses. If accepted, I strongly encourage the authors to respond to those concerns.

To be clear, I have my own concerns about this paper, which were not raised by the reviewers (for example, it seems like the "meta prompting" needed becomes a new set of static principles), but overall, I think that the contributions are interesting enough to disseminate to the research community.
Decision: Accept (poster)