Beyond In-Context Learning: Aligning Long-form Generation of Large Language Models via Task-Inherent Attribute Guidelines
Authors: Do Xuan Long, Duong Ngoc Yen, Do Xuan Trong, Anh Tuan Luu, Kenji Kawaguchi, Shafiq Joty, Min-Yen Kan, Nancy F. Chen
Accepted at: acl25

Beyond In-Context Learning: Aligning Long-form Generation of Large Language Models via Task-Inherent Attribute Guidelines

Do Xuan Long1,3, Duong Ngoc Yen2, Do Xuan Trong1*, Luu Anh Tuan2, Kenji Kawaguchi1, Shafiq Joty2,4, Min-Yen Kan1, Nancy F. Chen3
1National University of Singapore, 2Nanyang Technological University, Singapore, 3Institute for Infocomm Research (I2R), A*STAR, 4Salesforce AI Research
xuanlong.do@u.nus.edu

Abstract

In-context learning (ICL) is an important yet not fully understood ability of large language models (LLMs). It can greatly enhance task performance using a few examples, termed demonstrations, without fine-tuning. Although effective in question answering, ICL often underperforms in long-form generation tasks such as summarization. Under appropriately realistic assumptions, we empirically and theoretically show that ICL demonstrations alone are insufficient to teach LLMs the task’s language and format distributions for generation. We argue for explicit exposure to the task distributions and hypothesize that defining them by prompting enhances model performance. To this end, we present LongGuide, which efficiently generates two parallel streams of guidelines capturing task language and format properties: (i) Metric Guidelines (MGs) that instruct models to optimize self-evaluated metrics; and (ii) Output Constraint Guidelines (OCGs) that constrain generation at both token and sentence levels. LongGuide automatically selects the best combination of guidelines, improving both strong open- and closed-source LLMs by over 5% in both zero- and few-shot settings. LongGuide is generalizable, learnable by weak models to enhance strong ones, and integrates synergistically with automatic prompt optimizers1.

1 Introduction

In recent years, pre-trained large language models (LLMs) have demonstrated impressive instruction-based performance through zero- and few-shot learning capabilities (Brown et al., 2020; Chowdhery et al., 2022; OpenAI, 2022; Touvron et al., 2023; Jiang et al., 2023; Team et al., 2023). Notably, few-shot learning, termed as in-context learning (ICL), has proven highly effective and widely used to calibrate LLMs for applications (Dong et al., 2022). Formally, let \( \mathcal{V} \) be the vocabulary of the LM. For a test-time task \( T \), the goal is to generate a token sequence \( y \in \mathcal{Y} \subseteq \mathcal{V}^* \), given input token sequence \( x \in \mathcal{X} \subseteq \mathcal{V}^* \). Then, ICL generation using an LLM \( \mathcal{M} \) is the generation conditioned on \( x \) with \( k \) task demonstrations \( \{(x_1, y_1), ..., (x_k, y_k)\} \) concatenated into \( d_f = [x_1, y_1, ..., x_k, y_k] \in \mathcal{D} \subseteq \mathcal{V}^* \). The probability distribution induced from \( \mathcal{M}: \mathcal{V}^* \to \mathbb{R} \) is:

\[
P_{\mathcal{M}}(y|d_f, x) := \prod_{t=1}^{|\mathcal{Y}|} \mathcal{M}_{y^t}([x_1, y_1, \ldots, x_k, y_k, x, y^{<t}])
\]

where \( y = [y^1, \ldots, y^{|\mathcal{Y}|}] \) with \( y^t \in \mathcal{V} \). Several prior studies attempt to explain the ICL capabilities of LLMs, advocating for the sufficiency of

Dialogue: Morgan: Hey gorgeous, how's your day?
Suzanne: Nothing special, it's just one of many boring days at work. But... better...

Ground-truth: Suzanne is at work and is having a break now. Morgan invites Suzanne to a concert of Maroon 5 which takes place next week at the Hulu Theatre at Madison Square Garden. Suzanne agrees.

<table>
  <tr>
    <th>Without guideline</th>
    <th>With guideline</th>
  </tr>
  <tr>
    <td>
      <b>Zero-shot<br>(ROUGE-L = 25.00)</b><br>
      Output: The dialogue shows Morgan and Suzanne having a flirty conversation while discussing mundane work and expressing their affection for each other. Morgan surprises Suzanne with tickets to a Maroon 5 concert, and Suzanne expresses her excitement and gratitude.
    </td>
    <td>
      <b>Zero-shot + LongGuide<br>(MG-OCG, ROUGE-L = 29.85)</b><br>
      Guidelines: The summary must have from 1 to 4 sentences and from 5 to 51 words with an average of 22 words and 1 sentences.<br>
      Output: Morgan and Suzanne are talking about their day at work. Morgan offers Suzanne tickets to a Maroon 5 concert, and Suzanne expresses excitement and gratitude. They plan to attend the concert together.
    </td>
  </tr>
  <tr>
    <td>
      <b>Few-shot (3 random shots)<br>(ROUGE-L = 26.67)</b><br>
      Output: Morgan and Suzanne discuss their day, with Morgan inviting Suzanne to a Maroon 5 concert. Suzanne excitedly accepts and Morgan expresses his desire for her happiness.
    </td>
    <td>
      <b>Few-shot (3 shots) + LongGuide<br>(MG-OCG, ROUGE-L = 31.74)</b><br>
      Guidelines: The summary is highly concise and to the point without being comprehensive....The summary must have from 1 to 4 sentences and from 5 to 51 words with an average of 22 words and 1 sentences.<br>
      Output: Morgan invites Suzanne to a Maroon 5 event at Madison Square Garden and she happily accepts, expressing happiness and gratitude.
    </td>
  </tr>
</table>

Figure 1: ChatGPT results on a SAMSum example (Gliwa et al., 2019). With LongGuide, the generated output aligns better with ground truth, and the quality is also improved by removing verbose details: “The dialogue...” (ZS), "...discuss their day” (FS). See Appx.-Figure 14 for full texts.

1Works done during the internship at WING, NUS.
2Our codes and data will be made available at here.
well-chosen \( d_f \) as implicitly teaching the \( \mathcal{M} \) to perform the tasks, especially classification ones (Saunshi et al., 2020; Xie et al., 2021; Wang et al., 2024). Central to their theoretical analyses is a strong assumption that the language model \( \mathcal{M} \) fully captures the underlying distribution of the task’s language; i.e., \( P_{\mathcal{M}}(X) = P_T(X) \) where \( P_T \) is the task-specific data distribution. However, this assumption is often not met, especially with domain-specific terminologies (Cheng et al., 2024) (also see §C.1 for a case study), raising concerns about the actual sufficiency of ICL. Moreover, recent studies empirically show that ICL underperforms in long-form generation tasks involving multi-sentence or -paragraph answers (Sun et al., 2023a; Huang et al., 2024), highlighting significant gaps in our understandings of the causes of these limitations and how to effectively instruct LLMs for these tasks. These challenges remain unsolved to date.

In this work, we first study the proficiency of ICL for long-form generation tasks. We empirically and theoretically highlight that if a language model fails to capture the task’s *text properties (language and format)*, providing demonstrations alone with such properties cannot entirely resolve this (§2). This is because the model does not consistently apply them to all generated responses. Maintaining such properties in responses is crucial for accurately solving the task. Therefore, we argue that providing explicit task guidelines that capture these text properties is essential for improving LLM performance. Figure 1 illustrates such an example where instructing LLMs explicitly by guidelines carrying certain properties (e.g., conciseness, #sentences) of the task output distribution improves both alignment with ground truth and generation quality.

We then propose LongGuide (§3), a guideline-learning algorithm that efficiently\footnote{The prompts and cost analysis are discussed in §G.} generates two types of guidelines concurrently from limited task training data as supplementary instructions to enhance LLMs: (i) Metric Guidelines (MGs) that steer models to optimize self-evaluation guided metrics, inspired by prior studies in machine translation (Ranzato et al., 2015) and LLM self-evaluation (Ren et al., 2023); and (ii) Output Constraint Guidelines (OCGs) that impose constraints on generated outputs at the sentence and token levels, drawing on controllable generation research (Fan et al., 2018a). LongGuide is related to previous studies in task instruction construction (Wang et al., 2022b) and task understanding through task definitions (Yin et al., 2023). However, it differs by offering “post-hoc” instructions that guide LLMs to enhance responses based on learned quality and quantitative criteria.

LongGuide automatically identifies optimal guidelines, significantly enhancing distribution alignment and generation quality across seven generation tasks and one real-life chat LLM benchmark, including summarization, text simplification, translation, dialogue generation, and table-to-text generation. Its guidelines can enhance ICL performance through demonstrations (§5.2), improve non-instruct LLMs (§D.1), boost stronger models when learned by weaker ones (§D.2), and can be further optimized for usage using prompt optimization algorithms (§D.3). Notably, LongGuide is approximately at least 3.75 times more cost-efficient than prompt optimization algorithms (§G.3-Table 20) as it requires only four prompt variants to verify on the validation set while delivering superior performance.

2 ICL Alone Is Insufficient for Long-form Generation

A long-form generation task \( T \) with \( n \) samples is defined as \( D = \{(x_i^t, y_i^t)\}_{i=1}^n \) where \( x_i^t \) and \( y_i^t \) are input contexts and ground-truth sentence- or paragraph-long responses (Fan et al., 2019). For such tasks, preserving language and format properties of task-specific data during generation is essential for aligning outputs with ground truth. This is unlike classification, where outputs are predefined. We demonstrate that ICL fails to enable LLMs to maintain these properties during generation.

Setups. We first select metrics as properties commonly used for dialogue summarization. We follow Fu et al. (2023) to choose six: (1) Semantic Coverage (COV); (2) Factuality (FAC); (3) Consistency (CON); (4) Informativeness (INF); (5) Coherence (COH); (6) Relevance (REL). We also measure (7) # tokens (NT) and (8) # sentences (NS) of ICL responses, as these format metrics can significantly impact model performance (Fan et al., 2018a). *For each metric, we select the demonstrations having the same score and evaluate whether the ICL-generated responses maintain that score.*

Our experiments are performed on 100 random SAMSum samples (Gliwa et al., 2019) for each metric. We use ChatGPT (gpt-3.5-turbo-1106) (OpenAI, 2022) with Self-consistency (Wang et al.,
<table>
  <tr>
    <th>ICL w/ 5 demos</th>
    <th>(1) COV<br>Expected outcome 100%</th>
    <th>(2) FAC<br>100%</th>
    <th>(3) CON<br>100%</th>
    <th>(4) INF<br>100%</th>
    <th>(5) COH<br>100%</th>
    <th>(6) REL<br>100%</th>
    <th>(7) NT (mean)</th>
    <th>(7) NT (std)</th>
  </tr>
  <tr>
    <td>Mistral-7B-v0.3</td>
    <td>12%</td>
    <td>27%</td>
    <td>28%</td>
    <td>8%</td>
    <td>20%</td>
    <td>35%</td>
    <td>87.74</td>
    <td>144.91</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B</td>
    <td>12%</td>
    <td>42%</td>
    <td>50%</td>
    <td>4%</td>
    <td>32%</td>
    <td>47%</td>
    <td>271.81</td>
    <td>379.48</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>43%</td>
    <td><b>90%</b></td>
    <td><b>85%</b></td>
    <td><b>40%</b></td>
    <td>78%</td>
    <td><b>96%</b></td>
    <td>281.38</td>
    <td>264.59</td>
  </tr>
  <tr>
    <td>Mistral-7B-it-v0.2</td>
    <td>38%</td>
    <td>80%</td>
    <td>78%</td>
    <td>17%</td>
    <td>75%</td>
    <td>88%</td>
    <td>50.25</td>
    <td>55.54</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-it</td>
    <td><b>44%</b></td>
    <td>86%</td>
    <td>82%</td>
    <td>26%</td>
    <td><b>81%</b></td>
    <td>87%</td>
    <td><b>34.72</b></td>
    <td><b>45.29</b></td>
  </tr>
</table>

Table 1: % of responses scored 5 on the (1)-(6) metrics, and the (mean, std) of the (7) #tokens of the responses. Qwen scored high on metrics (1)-(6) because it copies the input dialogue as the summarization outcome.

2022a) to evaluate metrics (1)–(6) on a scale of 1–5, as it is an effective evaluator (Wang et al., 2023a). NLTK (Bird and Loper, 2004) assesses metrics (7)–(8). For ICL experiments, we examine five instruct and non-instruct models: Mistral-7B-v0.3 (Jiang et al., 2023), Llama-3.1-8B (Dubey et al., 2024), Qwen-2.5-7B (Qwen Team, 2024), ans Mistral-7B-it-v0.2 (Jiang et al., 2023) and Llama-3.1-8B-it (Dubey et al., 2024). For metrics (1)–(6), we select demonstrations having a perfect score of 5, and for metrics (7)–(8) having 17 response tokens, spanning 2 sentences. For each metric, we further examine whether a simple guideline: “The output must maintain...{property}.” (w/ guideline) can help instruct models (Mistral) maintain that property better during generation.

Findings. We present the \( k = 5 \) demonstration results in Table 1 and the case of Mistral-7B-it-v0.2 in Figure 7 with metrics (1)–(7)\footnote{We also tested demonstration counts of \( k = 3, 10 \), all follow similar trends, see Appendix C.3’s Figure 8.}. We derive three surprising findings. Firstly, the ICL models do not achieve a 100% score of 5 on any metric and instruct models generally outperform non-instruct models. The highest percentage of score 5 on average is on COH and REL, where ICL models already excel while for critical summarization metrics such as INF and COV, they achieve only up to 20% to 40%. Notably, although all demonstrations contain 17 output tokens, fewer than 5% answers achieve this property. Secondly, increasing # demonstrations does not rectify this issue; the same trends persist across 3, 5, and 10. Finally, by adding a simple guideline shown in Appx.Figure 7, the percentages of answers maintaining the metrics are mostly improved, especially (7) and (8), verifying that adding guidelines is indeed helpful for instruct models to maintain these properties. Without ICL (and without instruction in our consideration), the model is entirely unable to solve the task.

Theoretical intuitions. Our theoretical intuitions explaining the above observations are provided in §B. In summary, we prove that when an LLM does not fully capture the true task distribution (\( P_M \neq P_T \)), demonstrations cannot recover the true task distribution in the limit, causing certain task-specific language and format properties may not be preserved in generation. We term this as the text property transfer (PT) problem. To address this, we define a text property task as a reformulation of the original task, where responses are mapped via a property-specific metric. We hypothesize that the original task can be approximated by optimizing a set of well-chosen text property tasks, allowing the model to better align with desired text characteristics and improve generation quality.

3 LongGuide: An Efficient Guideline Generation Algorithm

Motivations. As we have seen, providing textual guidelines instructing LLMs to optimize certain text property metrics can enhance them on responses, possibly because LLMs are optimizers (Yang et al., 2024). We propose LongGuide (Figure 2 and Algorithm 1), an algorithm that efficiently generates guidelines for LLMs to optimize self-evaluated text properties during generation. Specifically, Steps 1–3 focus on generating the Metric Guideline (MG) capturing the intrinsic language properties of the task via reference-free metrics. In parallel, Step 4 analyzes the answer format of the task and translates it to Output Constraint Guideline (OCG). The best combination of MG and OCG is selected for inference (Step 5). To ensure LongGuide’s generalizability to new tasks, we assume access to at most 50 training samples:
\[
D^{train} = \{(x_i^t, y_i^t)\}_{i=1}^n.
\]

Step 1: Metric Collection & Selection. To learn a task’s language properties, this step reasons to select appropriate language evaluation metrics for
self-evaluation. For this purpose, we first construct a pool of evaluation metrics, \( S \), applicable to any text generation task. \( S \) consists of 27 distinct metrics from 4 main sources (Appx.-Table 13 for details). Specifically, we collect 3 metrics from ABC’s of Communication (Wagner, 1963), 12 metrics from (Fu et al., 2023) for dialogue generation, summarization, data2text generation, and machine translation, and propose 12 more metrics for a broader evaluation coverage. We do not collect LM-based metrics, such as FactScore (Min et al., 2023), because it is challenging for LLMs to define and self-evaluate them. Additionally, we do not gather definitions of collected metrics, as their interpretations may vary across different tasks.

With \( D^{train} \) and \( S \), we then perform \( K \) iterations to select the metrics. At each iteration, we randomly sample a batch of data from \( D^{train} \) and instruct \( \mathcal{M} \) to generate the top-5 most important metrics in \( S \) for evaluating batch data properties via Chain-of-Thought prompting (Wei et al., 2022). We implement the top-5 constraint to avoid excessive metrics being selected. The final set of selected metrics, denoted by \( M \), consists of the metrics chosen across all iterations sorted in alphabetic order.

Step 2: Metric Score Collection via Self-evaluation. This step focuses on evaluating the selected metrics from \( M \) on \( D^{train} \) to capture the task properties. Motivated by prior studies (Wang et al., 2023a; Ren et al., 2023), we utilize \( \mathcal{M} \) to score the metrics on a scale of 1–5. Specifically, for each train sample, \( \mathcal{M} \) scores its ground-truth answer on all \( M \)’s metrics via Self-Consistency (Wang et al., 2022a). The final metrics’ scores, denoted as \( scores_M \), are the average of scores over all train samples. Note that we separate this step from Step 1’s metric selection because we want to evaluate each chosen metric on \( D^{train} \) instead of the samples that led \( \mathcal{M} \) to select it.

Step 3: Generating Metric Guideline (MG). This step aims to generate a textual metric guideline (MG) that guides \( \mathcal{M} \) to align generation outputs with task-specific properties from \( scores_M \). MG is formed by concatenating metrics’ definitions generated by \( \mathcal{M} \) and tailored by \( scores_M \) via the LLM instruction “Based on these scores on a scale of 5...define the expected quality of the output for each metric in natural language”. We use these moderated definitions instead of raw \( scores_M \) because LLMs better capture contextual nuances through descriptions rather than numerical scores (Singh and Strouse, 2024). Figure 2 illustrates an instance where “Informativeness” in the task “dialogue sum.” achieving 4/5 score from Step 2 is defined as “…good amt. of informative…”.

Step 4: Output Constraint Guideline (OCG). Research on controllable generation has extensively proposed constraints including ones on the length, which are broadly applicable, as well as linguistic or keyword, which are more task-specific (Fan et al., 2018a; He et al., 2022). In this step, we aim to establish a robust set of output constraints that apply universally to long-form generation tasks. We focus on six key constraints related to two distributions: the number of sentences and tokens in ground-truth answers. These constraints include minimum, maximum, and average counts, serving as basic statistics for length and expected values. The Output Constraint Guideline (OCG) instructs

![Overview of LongGuide. Orange and blue boxes denote the learned metric guideline and output constraint guideline.](page_120_120_1208_370.png)

Figure 2: Overview of LongGuide. Orange and blue boxes denote the learned metric guideline and output constraint guideline.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">SAMSum</th>
    <th colspan="3">CNN</th>
    <th colspan="3">SWiPE</th>
  </tr>
  <tr>
    <th>R-L↑</th>
    <th>B-1↑</th>
    <th>BS↑</th>
    <th>Avg.JS↓</th>
    <th>R-L↑</th>
    <th>B-1↑</th>
    <th>BS↑</th>
    <th>Avg.JS↓</th>
    <th>R-L↑</th>
    <th>B-1↑</th>
    <th>BS↑</th>
    <th>Avg.JS↓</th>
  </tr>
  <tr>
    <td>Zero-shot (ZS)</td>
    <td>22.20</td>
    <td>20.05</td>
    <td>58.98</td>
    <td>0.1014</td>
    <td>19.23</td>
    <td>20.43</td>
    <td>60.59</td>
    <td>0.1262</td>
    <td>36.60</td>
    <td>39.01</td>
    <td><b>71.18</b></td>
    <td>0.0565</td>
  </tr>
  <tr>
    <td>+ OCG</td>
    <td><b>27.55</b></td>
    <td><b>28.64</b></td>
    <td>60.38</td>
    <td><b>0.0402</b></td>
    <td><b>22.46</b></td>
    <td><b>27.82</b></td>
    <td><b>61.37</b></td>
    <td><b>0.0718</b></td>
    <td><b>32.48</b></td>
    <td>32.88</td>
    <td>67.32</td>
    <td>0.0650</td>
  </tr>
  <tr>
    <td>+ MG</td>
    <td>27.81</td>
    <td><b>28.81</b></td>
    <td>60.06</td>
    <td>0.0388</td>
    <td>18.35</td>
    <td>19.66</td>
    <td>59.79</td>
    <td>0.1413</td>
    <td><b>38.21</b></td>
    <td><b>40.83</b></td>
    <td>70.87</td>
    <td><b>0.0550</b></td>
  </tr>
  <tr>
    <td>+ MG-OCG</td>
    <td>28.35</td>
    <td>28.79</td>
    <td>60.66</td>
    <td>0.0375</td>
    <td>22.05</td>
    <td>26.97</td>
    <td>61.18</td>
    <td>0.0789</td>
    <td>35.47</td>
    <td>36.95</td>
    <td>68.77</td>
    <td>0.0554</td>
  </tr>
  <tr>
    <td>+ LongGuide</td>
    <td><b>28.35</b></td>
    <td>28.79</td>
    <td><b>60.66</b></td>
    <td><b>0.0375</b></td>
    <td><b>22.46</b></td>
    <td><b>27.82</b></td>
    <td><b>61.37</b></td>
    <td><b>0.0718</b></td>
    <td><b>38.21</b></td>
    <td><b>40.83</b></td>
    <td><b>70.87</b></td>
    <td><b>0.0550*</b></td>
  </tr>
  <tr>
    <td>Few-shot (FS)</td>
    <td>27.13</td>
    <td>27.21</td>
    <td>61.70</td>
    <td>0.0502</td>
    <td>17.56</td>
    <td>20.55</td>
    <td>57.74</td>
    <td>0.0844</td>
    <td>39.47</td>
    <td>39.76</td>
    <td>70.56</td>
    <td>0.0469</td>
  </tr>
  <tr>
    <td>+ OCG</td>
    <td><b>27.84</b></td>
    <td><b>29.91</b></td>
    <td>61.08</td>
    <td><b>0.0336</b></td>
    <td><b>15.20</b></td>
    <td><b>17.58</b></td>
    <td><b>58.12</b></td>
    <td><b>0.0922</b></td>
    <td><b>29.54</b></td>
    <td><b>30.32</b></td>
    <td><b>68.82</b></td>
    <td><b>0.0596</b></td>
  </tr>
  <tr>
    <td>+ MG</td>
    <td>27.50</td>
    <td>30.15</td>
    <td>62.24</td>
    <td>0.0352</td>
    <td>18.13</td>
    <td>20.94</td>
    <td>57.89</td>
    <td>0.0830</td>
    <td><b>41.36</b></td>
    <td><b>41.22</b></td>
    <td><b>71.14</b></td>
    <td><b>0.0450</b></td>
  </tr>
  <tr>
    <td>+ MG-OCG</td>
    <td><b>30.65</b></td>
    <td><b>31.72</b></td>
    <td><b>62.73</b></td>
    <td><b>0.0318</b></td>
    <td><b>19.19</b></td>
    <td><b>22.30</b></td>
    <td><b>57.95</b></td>
    <td><b>0.0814</b></td>
    <td>38.56</td>
    <td>37.87</td>
    <td>68.54</td>
    <td>0.0529</td>
  </tr>
  <tr>
    <td>+ LongGuide</td>
    <td><b>30.65</b></td>
    <td><b>31.72</b></td>
    <td><b>62.73</b></td>
    <td><b>0.0318</b></td>
    <td><b>19.19</b></td>
    <td><b>22.30</b></td>
    <td><b>57.95</b></td>
    <td><b>0.0814</b></td>
    <td><b>41.36</b></td>
    <td><b>41.22</b></td>
    <td><b>71.14</b></td>
    <td><b>0.0450*</b></td>
  </tr>
</table>

Table 2: Mistral performance verifying LongGuide mitigates the text property transfer (PT) problem (\S2): (1) the trends of ROUGE-L (R-L), BLEU-1 (B-1), BERTScore (BS), and Jensen–Shannon divergence (Avg. JS) show strong correlations, supporting our hypothesis; (2) LongGuide substantially enhances Avg. JS scores, thereby mitigating the PT problem.

\( \mathcal{M} \) to adhere to these statistics during generation.

Step 5: MG–OCG selection. Models have varying inherent knowledge per task, resulting in different improvements gained by using MG and OCG (Table 4). This step determines the optimal MG and OCG combination by evaluating model performance on \( D^{train} \) across configurations. The best-performing guideline configuration is then selected as the final output of LongGuide.

4 Experiments

Benchmarks. We benchmark LongGuide on seven widely evaluated long-form generation tasks from four main categories, summarization, text simplification, machine translation and generation, and one real-life chat LLM benchmark. These tasks are SAMSum (Gliwa et al., 2019), CNN/Daily Mail (3.0.0) (See et al., 2017) and XL-SUM (Hasan et al., 2021) for summarization, SWiPE (Laban et al., 2023) for text simplification, IWSLT-2017 en-ja (Cettolo et al., 2017) for machine translation, Synthetic-Persona-Chat (Jandaghi et al., 2023) for dialogue generation, CommonGen-Challenge (Lin et al., 2020) for data-to-text generation, and (a subset of) AlpacaEval2 (Dubois et al., 2024). We also benchmark the reasoning tasks in \S E.3. See \S F for details.

Baselines and evaluations. Since LongGuide is the first method to self-learn guidelines as additional instructions for generation, we compare it with the zero-/few-shot prompting baselines in this section, and many-shot prompting in \S E.1. We also evaluate it against three of the strongest prompt optimization algorithms to date: APO (Pryzant et al., 2023) in this section, and EvolPrompt (Guo et al., 2024) and adv-ICL (Do et al., 2024) in \S D.3, both of which optimize the input prompt on the \( D^{train} \). We also compare LongGuide with “General Guidelines” in \S 5.2 where we ask the models to reason over demonstrations to generate task guidelines. For models, we empirically examine both strong open- and closed-source LLMs: Mistral-7B-it v0.2 (Jiang et al., 2023) as an open-source model and ChatGPT (gpt-3.5-turbo-1106) (OpenAI, 2022) as a closed-source model. For evaluations, we use ROUGE-L (Lin, 2004) (recall-based) following Bai et al. (2024) (also for LongGuide’s Step 5), and GPT-4o-Judge (OpenAI, 2024) as our main evaluation metrics. For GPT-4o-Judge, we evaluate how aligned the generated answer is with the reference answer and its quality on five criteria: (i) Format consistency; (ii) Content completeness; (iii) Factuality; (iv) Style adherence; (v) Generation quality on a scale of 1–10, following Zheng et al. (2023) (see \S G.1 for full prompt). We also report BLEU-1 (Papineni et al., 2002) (precision-based), BERTScore (Zhang et al., 2020) (meaning-based), and Human evaluation verifying the metric optimization and generation quality in \S 5.1. The results are averaged over three runs, with 95% CI of the t-test.

4.1 Findings

LongGuide enhances PT which correlates with improved performance. LongGuide effectively addresses the PT problem identified in \S 2. Our experiments are conducted on 3 benchmarks SAMSum, CNN, and SWiPE with Mistral under the zero-shot and few-shot settings. For each task, we first obtain the set of selected text properties from LongGuide that the model needs to optimize. We then measure the average of Jensen-Shannon divergence (Lin, 1991) between their score distributions (judged by ChatGPT) between the generated answers and the ground truth answers, across all
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">Sum.</th>
    <th colspan="3">Simplification</th>
    <th colspan="3">Translation</th>
    <th colspan="3">Dialogue Gen.</th>
    <th colspan="3">Table2Text</th>
  </tr>
  <tr>
    <th>SAMSum</th>
    <th>CNN (3.0.0)</th>
    <th>XL-Sum</th>
    <th>SWIPE</th>
    <th>IWSLT17 en-ja</th>
    <th>Syn. Persona</th>
    <th>Comm.-Chall.</th>
    <th>Syn. Persona</th>
    <th>Comm.-Chall.</th>
    <th>Syn. Persona</th>
    <th>Comm.-Chall.</th>
    <th>Syn. Persona</th>
    <th>Comm.-Chall.</th>
  </tr>
  <tr>
    <td>#shots (ran.)</td>
    <td>3</td>
    <td>3</td>
    <td>5</td>
    <td>3</td>
    <td>3</td>
    <td>5</td>
    <td>5</td>
    <td>5</td>
    <td>5</td>
    <td>5</td>
    <td>5</td>
    <td>5</td>
    <td>5</td>
  </tr>
  <tr>
    <td>Zero-shot</td>
    <td>22.20 / 7.43</td>
    <td>19.23 / 7.38</td>
    <td>9.19 / 5.96</td>
    <td>36.60 / 7.21</td>
    <td>13.12 / 2.82</td>
    <td>12.76 / 2.68</td>
    <td>10.12 / 5.14</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>+ APO</td>
    <td>23.77 / 7.31</td>
    <td>19.53 / 7.40</td>
    <td>12.06 / 5.85</td>
    <td>36.92 / 7.21</td>
    <td>14.45 / 2.91</td>
    <td>10.66 / 2.41</td>
    <td>11.21 / 4.68</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td><i>LongGuide</i></td>
    <td><b>28.35 / 7.73</b></td>
    <td><b>22.46 / 7.45</b></td>
    <td><b>13.38 / 6.29</b></td>
    <td><b>38.21 / 7.32</b></td>
    <td><b>16.35 / 3.45</b></td>
    <td><b>14.67 / 4.45</b></td>
    <td><b>25.20 / 6.81</b></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>% gain (+)</td>
    <td>6.15 / 0.30</td>
    <td>3.23 / 0.07</td>
    <td>5.19 / 0.33</td>
    <td>1.61 / 0.11</td>
    <td>3.41 / 0.63</td>
    <td>1.93 / 1.77</td>
    <td>15.08 / 1.67</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Mistral-10 (v.2)</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Few-shot</td>
    <td>27.13 / 7.66</td>
    <td>17.56 / 5.84</td>
    <td>9.79 / 4.46</td>
    <td>39.47 / 7.12</td>
    <td>12.69 / 2.66</td>
    <td>3.56 / 1.00</td>
    <td>3.98 / 1.34</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>+ APO</td>
    <td>26.23 / 7.44</td>
    <td>18.18 / 5.89</td>
    <td>11.99 / 4.55</td>
    <td>39.55 / 7.11</td>
    <td>14.08 / 2.92</td>
    <td>4.26 / 1.05</td>
    <td>5.45 / 2.05</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td><i>LongGuide</i></td>
    <td><b>30.65 / 7.72</b></td>
    <td><b>19.19 / 7.59</b></td>
    <td><b>15.23 / 5.06</b></td>
    <td><b>41.36 / 7.24</b></td>
    <td><b>16.62 / 3.40</b></td>
    <td><b>5.25 / 3.95</b></td>
    <td><b>25.05 / 6.65</b></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>% gain (+)</td>
    <td>3.52 / 0.06</td>
    <td>1.63 / 0.15</td>
    <td>5.44 / 0.40</td>
    <td>1.89 / 0.12</td>
    <td>3.66 / 0.74</td>
    <td>1.69 / 2.93</td>
    <td>21.07 / 5.31</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ChatGPT</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Zero-shot</td>
    <td>23.83 / 7.43</td>
    <td>20.12 / 7.44</td>
    <td>10.80 / 5.96</td>
    <td>45.09 / 7.28</td>
    <td>36.13 / 7.62</td>
    <td>19.46 / 6.04</td>
    <td>24.21 / 6.53</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>+ APO</td>
    <td>25.05 / 7.45</td>
    <td>20.34 / 7.39</td>
    <td>12.19 / 6.07</td>
    <td><b>46.32 / 7.51</b></td>
    <td>37.74 / 7.44</td>
    <td>19.91 / 6.12</td>
    <td>23.63 / 6.53</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td><i>LongGuide</i></td>
    <td><b>30.47 / 7.59</b></td>
    <td><b>22.19 / 7.67</b></td>
    <td><b>20.93 / 6.56</b></td>
    <td><b>45.09 / 7.28</b></td>
    <td><b>41.27 / 8.11</b></td>
    <td><b>22.98 / 7.41</b></td>
    <td><b>34.41 / 7.23</b></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>% gain (+)</td>
    <td>6.64 / 0.16</td>
    <td>2.07 / 0.23</td>
    <td>10.13 / 0.40</td>
    <td>0.00 / 0.00</td>
    <td>5.09 / 0.49</td>
    <td>3.52 / 0.37</td>
    <td>10.20 / 0.70</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Few-shot</td>
    <td>22.21 / 7.32</td>
    <td>14.51 / 4.38</td>
    <td>11.42 / 5.95</td>
    <td>33.72 / 5.07</td>
    <td>31.93 / 7.25</td>
    <td>16.10 / 4.67</td>
    <td>22.08 / 4.19</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>+ APO</td>
    <td>24.22 / 7.28</td>
    <td>15.20 / 4.01</td>
    <td>14.07 / 6.19</td>
    <td>34.46 / 5.13</td>
    <td>33.72 / 7.31</td>
    <td>17.68 / 4.55</td>
    <td>25.09 / 6.12</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td><i>LongGuide</i></td>
    <td><b>31.46 / 7.72</b></td>
    <td><b>18.17 / 4.22</b></td>
    <td><b>19.95 / 6.36</b></td>
    <td><b>37.60 / 7.25</b></td>
    <td><b>38.37 / 7.91</b></td>
    <td><b>22.56 / 7.26</b></td>
    <td><b>38.21 / 7.21</b></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>% gain (+)</td>
    <td>9.25 / 0.40</td>
    <td>3.66 / 0.04</td>
    <td>8.53 / 0.41</td>
    <td>3.88 / 0.18</td>
    <td>6.50 / 0.66</td>
    <td>6.53 / 0.59</td>
    <td>16.13 / 3.02</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 3: ROUGE-L / GPT-4o-Judge results on seven long-form generation tasks. LongGuide remarkably outperforms baselines on most tasks and substantially enhances LLMs. BLUE-1 scores are reported in Appx.-10.

selected properties, denoted as Avg.JS.

The results are presented in Table 2. LongGuide significantly lowers the Avg.JS scores compared to the baselines, demonstrating the effectiveness of guidelines for enhancing property transfer. Furthermore, our findings corroborate our hypothesis: across benchmarks, Avg.JS exhibits moderate to strong positive correlations with the performance metrics (ROUGE-L, BLEU-1, BERTScore) measured by the Pearson correlation coefficient (Pearson, 1895) (Appx.-Figure 9). In §E.7, we present the density plots for all metrics measured on the results with and without LongGuide.

Table 3 details our main experiment results on downstream tasks. Firstly, for baselines, zero-shot performance is interestingly higher than the few-shot for both models, and the gaps are huge for Synthetic Persona and CommonGen-Challenge. We hypothesize that models were partly exposed to task data during training, causing few-shot demonstrations to push the prompts out of distribution, leading to frequent refusals to answer. Meanwhile, LongGuide helps models overcome this issue. Secondly, LongGuide substantially improves zero- and few-shot baselines by 6% on ROUGE-L and 0.8 on GPT-4o-Judge on average: improvement for few-shot prompting is surprisingly higher than in zero-shot, possibly because improving a stronger baseline is harder than a weaker one. Notably, LongGuide outperforms APO in most benchmarks, especially under the zero-shot setting, demonstrating that our strategy of optimizing text property tasks is markedly more effective than APO optimizing only ROUGE-L on limited data. Thirdly, we observe that LongGuide achieves the highest improvements on CommonGen-Challenge with 15.62% and lowest on SWIPE with 1.85% on ROUGE-L. These improvements are mainly because the answers generated by the baselines are often far longer than the ground truth. LongGuide rectifies this issue by controlling the output length and quality, leading to substantial performance gains. Fourthly, among the two models, LongGuide interestingly improves Mistral by 5.39%, while ChatGPT, regarded as stronger, improves by a larger margin, 6.58%. This suggests that LongGuide has the potential to benefit stronger models in the future. Among five GPT-4o-Judge criteria in Figure 3, LongGuide notably improves Format, Style, and Factuality, confirming its effectiveness in aligning model generation with ground-truth distributions. Finally, the significant gains in Quality, together with the ROUGE-L scores from Table 3 further verify that LongGuide also strongly enhances the generation quality.

Where do the enhancements come from? To identify the primary source of performance gains, we present the results of LLMs with LongGuide’s components in Table 4. Firstly, MG-OCG combination (w/ MG-OCG) is the most useful for LLMs, observed to be the best 15 times, followed by OCG

![GPT-4-Judge scores over criteria.](page_1012_1342_388_246.png)

Figure 3: GPT-4-Judge scores over criteria.
<table>
  <tr>
    <th>Method</th>
    <th>SAMSum</th>
    <th>CNN (3.0)</th>
    <th>XL-Sum</th>
    <th>SWiPE</th>
    <th>IWSLT17 en-ja</th>
    <th>Synthetic Persona</th>
    <th>CommGen-Chall.</th>
  </tr>
  <tr>
    <td>Zero-shot (ZS)</td>
    <td>22.20±0.44</td>
    <td>19.23±0.34</td>
    <td>9.19±0.93</td>
    <td>36.60±1.59</td>
    <td>13.12±1.39</td>
    <td>12.76±1.54</td>
    <td>10.12±0.82</td>
  </tr>
  <tr>
    <td>+ OCG</td>
    <td>27.55±0.97</td>
    <td><b>22.46±0.64</b></td>
    <td><b>14.38±0.17</b></td>
    <td><b>37.48±1.17</b></td>
    <td><b>16.53±0.79</b></td>
    <td><b>13.35±0.47</b></td>
    <td><b>23.16±1.17</b></td>
  </tr>
  <tr>
    <td>+ MG</td>
    <td>27.81±1.17</td>
    <td>18.35±0.60</td>
    <td>9.37±0.27</td>
    <td><b>38.21±1.72</b></td>
    <td>8.71±0.33</td>
    <td>12.53±0.51</td>
    <td>21.54±2.07</td>
  </tr>
  <tr>
    <td>+ MG-OCG</td>
    <td><b>28.35±1.66</b></td>
    <td>22.05±0.84</td>
    <td>13.64±0.38</td>
    <td>35.47±2.89</td>
    <td>15.76±1.85</td>
    <td><b>14.69±1.08</b></td>
    <td><b>25.20±1.89</b></td>
  </tr>
  <tr>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
  </tr>
  <tr>
    <td>Few-shot (FS)</td>
    <td>27.13±0.26</td>
    <td>17.56±0.63</td>
    <td>9.79±0.18</td>
    <td>39.47±0.42</td>
    <td>12.69±1.82</td>
    <td>3.56±0.36</td>
    <td>3.98±0.17</td>
  </tr>
  <tr>
    <td>+ OCG</td>
    <td>27.84±0.77</td>
    <td><b>17.50±0.57</b></td>
    <td><b>17.22±1.17</b></td>
    <td><b>29.54±1.30</b></td>
    <td><b>16.62±0.77</b></td>
    <td><b>5.06±0.78</b></td>
    <td><b>25.05±0.78</b></td>
  </tr>
  <tr>
    <td>+ MG</td>
    <td>27.50±0.85</td>
    <td>18.13±0.28</td>
    <td>11.80±0.66</td>
    <td><b>41.36±1.37</b></td>
    <td>8.67±0.62</td>
    <td>4.32±0.30</td>
    <td>14.58±2.41</td>
  </tr>
  <tr>
    <td>+ MG-OCG</td>
    <td><b>30.65±0.88</b></td>
    <td><b>19.19±0.49</b></td>
    <td><b>15.23±0.33</b></td>
    <td>38.56±1.39</td>
    <td>15.83±0.95</td>
    <td><b>5.25±0.94</b></td>
    <td><b>5.94±1.00</b></td>
  </tr>
  <tr>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
  </tr>
  <tr>
    <td>Zero-shot (ZS)</td>
    <td>23.83±0.24</td>
    <td>20.12±0.27</td>
    <td>10.80±0.18</td>
    <td><b>45.09±1.42</b></td>
    <td>36.13±0.87</td>
    <td>19.46±1.40</td>
    <td>24.21±0.77</td>
  </tr>
  <tr>
    <td>+ OCG</td>
    <td>29.19±0.77</td>
    <td><b>22.39±0.87</b></td>
    <td><b>20.93±0.74</b></td>
    <td>37.76±1.47</td>
    <td>38.86±1.17</td>
    <td><b>22.98±1.07</b></td>
    <td><b>33.41±1.07</b></td>
  </tr>
  <tr>
    <td>+ MG</td>
    <td>25.38±0.70</td>
    <td>20.37±0.41</td>
    <td>10.42±1.14</td>
    <td>45.06±2.96</td>
    <td>37.88±2.42</td>
    <td>19.91±1.59</td>
    <td>17.23±2.77</td>
  </tr>
  <tr>
    <td>+ MG-OCG</td>
    <td><b>30.47±1.57</b></td>
    <td><b>22.19±0.65</b></td>
    <td><b>20.02±0.89</b></td>
    <td>41.38±4.91</td>
    <td><b>41.22±0.46</b></td>
    <td>20.95±1.01</td>
    <td>31.57±0.93</td>
  </tr>
  <tr>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
  </tr>
  <tr>
    <td>Few-shot (FS)</td>
    <td>22.21±0.26</td>
    <td>14.51±0.60</td>
    <td>11.42±0.13</td>
    <td>33.72±1.31</td>
    <td>31.92±1.88</td>
    <td>16.10±2.61</td>
    <td>22.08±0.69</td>
  </tr>
  <tr>
    <td>+ OCG</td>
    <td>27.00±1.73</td>
    <td><b>18.17±1.73</b></td>
    <td><b>17.95±1.73</b></td>
    <td>16.68±2.01</td>
    <td>38.57±1.37</td>
    <td><b>22.36±1.73</b></td>
    <td><b>38.12±1.73</b></td>
  </tr>
  <tr>
    <td>+ MG</td>
    <td>29.43±1.85</td>
    <td>15.45±2.16</td>
    <td>12.49±0.59</td>
    <td>19.36±1.40</td>
    <td><b>39.45±3.55</b></td>
    <td>18.64±1.49</td>
    <td>22.18±2.56</td>
  </tr>
  <tr>
    <td>+ MG-OCG</td>
    <td><b>31.46±1.34</b></td>
    <td><b>14.84±2.58</b></td>
    <td><b>18.58±0.44</b></td>
    <td><b>37.60±2.85</b></td>
    <td>38.43±2.37</td>
    <td>19.47±2.20</td>
    <td><b>38.21±3.70</b></td>
  </tr>
  <tr>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
    <td><i>MG-OCG</i> (0.2)</td>
  </tr>
</table>

Table 4: ROUGE-L results with 95% CI from t-test. The gains of LongGuide’s components vary across different models and tasks. The “MG-OCG selection” results are reported in Appx.-Table 14.

(w/ OCG) with 10, and MG (w/ MG) twice. While these statistics underscore the effectiveness of MG-OCG, OCG particularly proves itself highly effective in tasks such as summarization, translation, and table-to-text generation. Secondly, individual MG or OCG strengthens the prompting baselines, with OCG showing a slight edge. This is because while MG focuses on the language properties of answers, it does not directly control the output format, sometimes causing longer/shorter answers than the ground truths. Exceptionally, on SWiPE, OCG harms all models, whereas MG shows notably strong effectiveness with Mistral. Manual investigations reveal that ground-truth answers in SWiPE exhibit high variances in #sentences and #tokens which explains why OCG may not be effective for this benchmark. Thirdly, an interesting case is ChatGPT with few-shot prompting on SWiPE, where individual MG and OCG impair performance but their combination enhances it. This shows evidence that MG and OCG complement each other. As discussed above, due to the uneven nature of answers in SWiPE, using MG or OCG alone may not work well for multiple samples, as MG and OCG only provide expected statistics. However, combining them could enhance performance by allowing them to complement each other. A such complement SWiPE example is outlined in Appx.-Figure 16.

5 Discussion

We address several key questions about the usefulness, applicability, and generalizability of LongGuide. Additional properties are provided in §D and more method and attention analyses in §E.

5.1 Human Evaluation: Does LongGuide Enhance Generation Quality?

![Win/Draw/Loss rates of w/ versus w/o LongGuide.](page_1012_1342_484_246.png)

Figure 4: Win/Draw/Loss rates of w/ versus w/o LongGuide.

We perform a human evaluation to quantify whether LongGuide helps LLMs optimize the selected metrics and enhance generation quality, as no automatic methods can address this need to date. For this purpose, we randomly select 50 zero-shot generated samples from the SAMSum and Syn. Persona (since MG-OCG is the best for these datasets, Table 4). Three excellent English-native undergrads are hired to rate whether ZS + LongGuide improves ZS on each of the selected MG and OCG metrics. Due to limited resources, we evaluate 5 random MG metrics.

As shown in Figure 4, we notice that ZS + LongGuide outperforms ZS on 27.8% MG metrics on average, draws on 64.2%, and loses on only 8%. Specifically, among the MG metrics, “Brevity” shows the highest winning rate of 73% while “Relevance” obtains the lowest winning rate of 12%,
possibly because ZS models can already generate highly relevant outcomes. Meanwhile, on the OCG metrics, LongGuide achieves a superior win of 95% on average. Finally, regarding the generation quality, our annotators prefer LongGuide output by up to 92%. These indicate that LongGuide not only aligns the outputs with the ground truths but also enhances the generation quality. The fine-grained scores of MG metrics are provided in §E.10, and annotator agreement (Krippendorff’s alpha (Krippendorff, 2022)) is \( \alpha = 68.9\% \).

5.2 LongGuide Learns From Demonstrations To Boost ICL Performance

Here, we revisit the question posed in §2 and verify that LongGuide learned from demonstrations substantially increases ICL performance. Our experiments using Mistral cover CNN, IWSLT17 en-ja, and CommGen-Chall datasets. Our experiments involve averaging the performance under zero- and few-shot settings. For Baseline, no guideline is utilized. For LongGuide on Demos, we train LongGuide on demonstrations used in Table 3, in contrast to the \( D^{train} \) for the case of LongGuide. We add one more baseline, General Guidelines (Gen. Gui.) on Demos, where we ask the models to generate general task guidelines from demonstrations.

The results are summarized in Figure 5, with details in Appx.-Table 15. Specifically, LongGuide trained on \( D^{train} \) outperforms it on demonstrations, suggesting its possible scalability with more training data. Moreover, while Gen. Gui. slightly worsens the Baseline on CNN, both LongGuide and LongGuide on Demos notably surpass the Baseline, and Gen. Gui., highlighting the effectiveness of LongGuide in capturing task-specific properties, thereby boosting ICL performance.

5.3 Ablation Studies of LongGuide’s Steps

From Table 4, we identify the unique contributions of each step within LongGuide. Specifically, omitting Step 1 transforms it into OCG, whereas excluding Step 3 yields MG, and skipping Step 4 becomes MG-OCG. Here, we investigate LongGuide effectiveness when skipping Step 2, Metrics’ scores collection: for selected metrics from Step 1, we directly task the models to optimize them for the generated answers. We experiment with Mistral on SAMSum, SWiPE, and CommGen-Chall. datasets because for these datasets, the best guideline combination includes MG.

The results in Table 5 verify that without Step 2, the model performs worse, particularly for SAMSum and SWiPE in the zero-shot setting. We attribute these drops to the incorrect task properties captured and metric conflicts. A case study is provided in Appx.-Figure 18.

<table>
  <tr>
    <th>Methods</th>
    <th>SAMSum</th>
    <th>SWiPE</th>
    <th>CommGen-Chall.</th>
  </tr>
  <tr>
    <td>Zero-shot (ZS)</td>
    <td>22.20 / 7.43</td>
    <td>36.60 / 7.21</td>
    <td>10.12 / 5.14</td>
  </tr>
  <tr>
    <td>+ <b>LongGuide</b></td>
    <td><b>28.35 / 7.73</b></td>
    <td><b>38.21 / 7.32</b></td>
    <td><b>25.20 / 6.81</b></td>
  </tr>
  <tr>
    <td>+ LongGuide w/o step 2</td>
    <td>26.99 / 7.49</td>
    <td>36.90 / 7.22</td>
    <td>25.03 / 6.66</td>
  </tr>
  <tr>
    <td>Few-shot (FS)</td>
    <td>27.13 / 7.66</td>
    <td>39.47 / 7.12</td>
    <td>3.98 / 1.34</td>
  </tr>
  <tr>
    <td>+ <b>LongGuide</b></td>
    <td><b>30.65 / 7.72</b></td>
    <td><b>41.36 / 7.24</b></td>
    <td><b>25.05 / 6.65</b></td>
  </tr>
  <tr>
    <td>+ LongGuide w/o step 2</td>
    <td>30.37 / 7.70</td>
    <td>35.54 / 6.28</td>
    <td>24.15 / 5.82</td>
  </tr>
</table>

Table 5: Mistral ROUGE-L / GPT-4o-Judge main ablation study with LongGuide when Step 2 is skipped.

<table>
  <tr>
    <th>Methods</th>
    <th>LC Win Rate</th>
    <th>Win Rate</th>
  </tr>
  <tr>
    <td>Zero-shot (ZS)</td>
    <td>11.08%</td>
    <td>3.17%</td>
  </tr>
  <tr>
    <td>+ OCG</td>
    <td>4.73%</td>
    <td>2.44%</td>
  </tr>
  <tr>
    <td>+ MG</td>
    <td><b>19.13%</b></td>
    <td><b>7.07%</b></td>
  </tr>
  <tr>
    <td>+ MG-OCG</td>
    <td>8.42%</td>
    <td>3.90%</td>
  </tr>
  <tr>
    <td>+ <b>LongGuide</b></td>
    <td><b>19.13%</b></td>
    <td><b>7.07%</b></td>
  </tr>
  <tr>
    <td>Few-shot (FS)</td>
    <td>8.08%</td>
    <td>2.68%</td>
  </tr>
  <tr>
    <td>+ OCG</td>
    <td>7.73%</td>
    <td>3.45%</td>
  </tr>
  <tr>
    <td>+ MG</td>
    <td><b>12.65%</b></td>
    <td><b>4.88%</b></td>
  </tr>
  <tr>
    <td>+ MG-OCG</td>
    <td>12.63%</td>
    <td>4.88%</td>
  </tr>
  <tr>
    <td>+ <b>LongGuide</b></td>
    <td><b>12.65%</b></td>
    <td><b>4.88%</b></td>
  </tr>
</table>

Table 6: AlpacaEval2 experiments.

Figure 5: LongGuide learned from demonstrations substantially enhances Mistral performance (ROUGE-L).

Table 6 presents our findings. Few-shot demon-
strations and OCG negatively impact performance, likely due to the OOD nature of Alpaca-GPT4 compared to AlpacaEval2. In contrast, with just 5 Alpaca-GPT4 samples, MG metrics, and LongGuide enhance performance by capturing certain response properties from GPT-4 (OpenAI, 2023), nearly doubling the zero-shot points.

6 Related Work

Automatic prompt design for long-form generation. Long-form generation tasks are essential and have been studied extensively (Li et al., 2024). With LLM advancements, adapting these models for such tasks using prompt-based methods is critical yet challenging. Previous studies (Bang et al., 2023; Yang et al., 2023a; Hadi et al., 2023; Zhou et al., 2023b; Pan et al., 2024) highlight the limited efficacy of LLMs in producing outputs that resemble ground truths, as evaluated by ROUGE-L (Lin, 2004). Our approach autonomously composes supplementary contexts, integrating text evaluation metrics and format constraints. In addition, studies regarding enhancing instructions for LLMs (Wang et al., 2022b; Yin et al., 2023; Long et al., 2024; Wang et al., 2023b), automatic prompt optimization (Zhou et al., 2023a; Pryzant et al., 2023), and demonstration selection (Yang et al., 2023b; Qin et al., 2023) are also related areas that can be developed in parallel and combined with our work (§D.3).

Controllable generation with LLMs. Controllable generation during fine-tuning has been studied extensively (Fan et al., 2018a; Lakew et al., 2019; Martin et al., 2020; He et al., 2022). More recently, researchers have explored prompting methods to control LLM generation. For instance, Sun et al. (2023b) found that LLMs struggle to meet fine-grained hard constraints, while Fonseca and Cohen (2024) proposed controlling stylistic features like keywords and narrative during generation, leading to improved LLM summarization outcomes. Although (Lu et al., 2023; Fonseca and Cohen, 2024) are closely related to our OCG, our approach goes beyond summarization and open-ended only features, as discussed in §3. We focus on universally applicable features.

7 Conclusion

In this paper, we demonstrate that in-context learning (ICL) falls short in implicitly ensuring that large language models (LLMs) consistently preserve essential language and format properties in long-form generation tasks. To address this challenge, we introduce LongGuide, an efficient algorithm that automatically learns the critical language and format properties from task-specific data, converting them into textual guidelines for LLMs. Our results show that LongGuide significantly improves LLM performance across seven generation tasks and is highly generalizable, offering strong potential for various downstream applications with minimal data. This work paves the way for adaptive, task-specific prompt generation, advancing LLM adaptation.

Generalizability and Customization of LongGuide

LongGuide facilitates flexible generalization that allows customization and extension of guidelines MG and OCG for specific tasks, which we strongly recommend. For instance, in summarization, MG can focus on only 4-5 standard metrics from \( S \) while integrating summary-specific metrics like “Summary Structure” and “Retention of Core Supporting Evidence.” Simultaneously, OCG can impose stricter constraints on topics, keywords, grammar, or tones (Fan et al., 2018a; Lakew et al., 2019; Martin et al., 2020). Although LongGuide is primarily presented for general long-form generation, we strongly advise for these customizations to enhance its effectiveness.

Limitations

Our study has several limitations. Firstly, our theoretical analysis focuses solely on the task language distribution which is \( P_M(X) \) or \( P_M(X|D_f) \) instead of the actual output distribution, which is \( \arg\max_{y \in Y} P_M(Y = y \mid X) \) or \( \arg\max_{y \in Y} P_M(Y = y \mid D_f, X) \). In our study, while leveraging the task language distribution allows us to hypothesize and highlight the limitations of demonstrations, shifting focus to the actual output distribution could yield more insights.

Secondly, LongGuide’s learned guidelines are based on task-level and average statistics rather than sample-based details. We designed our framework at the task level to address limited data constraints, as we found that sample-based learning under these conditions leads to high errors. While task-level guidelines already demonstrate significant improvements for LLMs, sample-based guidelines could offer more tailored guidance, poten-
tially leading to optimal results. Moreover, this average guidance approach may be ineffective for tasks with high variance in the statistics that LongGuide learns. In such cases, the final step of LongGuide can prevent performance decline by likely choosing no guideline. For example, we found this applies to Code2Text (Richardson et al., 2017) & StoryGeneration (Fan et al., 2018b).

Thirdly, LongGuide relies on models having a certain level of task knowledge to perform self-evaluation effectively, and LongGuide necessitates LLMs with strong instruction-following capabilities. However, we anticipate that cutting-edge AI language models will overcome this limitation both now and in the near future.

Lastly, the guidelines learned by LongGuide may not be useful for the tasks the models are trained on. This is because these guidelines might introduce out-of-distribution context relative to the training data, thereby reducing the effectiveness of the testing inference. For instance, while we see notable enhancements on the CommonGen-Challenge dataset (Lin et al., 2020), it’s intriguing that we don’t observe any improvements on the WebNLG (Gardent et al., 2017) and E2E NLG (Puzikov and Gurevych, 2018) datasets, despite their expected similarity. Given the popularity of these datasets, we suspect the models we tested may have been previously trained on them.

Ethical Considerations

This method could be misused to optimize prompts for harmful purposes such as generating misinformation, hate speech, or privacy violations. While our method is not intended for such uses, it is impossible to completely prevent misuse. Although our method could enhance the efficiency and efficacy of bad actors, we do not anticipate that LongGuide is inherently more effective in these negative contexts than in positive applications. Finally, we employ annotators at an hourly rate of $20, which exceeds the local minimum wage requirements.

Acknowledgement

This research is supported by the National Research Foundation Singapore under the AI Singapore Programme (AISG Award No: AISG2-GC-2022-005, AISG Award No: AISG2-TC2023-010-SGIL) and the Singapore Ministry of Education Academic Research Fund Tier 1 (Award No: T1 251RES2207). DXL is supported by the A*STAR Computing and Information Science (ACIS) scholarship. We thank members of WING and Deep Learning Lab at NUS and the ACL RR anonymous reviewers for the constructive feedback.

References

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongBench: A bilingual, multi-task benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3119–3137, Bangkok, Thailand. Association for Computational Linguistics.

Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675–718, Nusa Dua, Bali. Association for Computational Linguistics.

Steven Bird and Edward Loper. 2004. NLTK: The natural language toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pages 214–217, Barcelona, Spain. Association for Computational Linguistics.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Niehues, Sebastian Stüker, Katsuhito Sudoh, Koichiro Yoshino, and Christian Federmann. 2017. Overview of the IWSLT 2017 evaluation campaign. In Proceedings of the 14th International Conference on Spoken Language Translation, pages 2–14, Tokyo, Japan. International Workshop on Spoken Language Translation.

Daixuan Cheng, Shaohan Huang, and Furu Wei. 2024. Adapting large language models via reading comprehension. In The Twelfth International Conference on Learning Representations.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao,
Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaeck Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catàsta, Jason Wei, Kathleen S. Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language modeling with pathways. J. Mach. Learn. Res., 24:240:1–240:113.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

Xuan Long Do, Yiran Zhao, Hannah Brown, Yuxi Xie, James Xu Zhao, Nancy F. Chen, Kenji Kawaguchi, Michael Shieh, and Junxian He. 2024. Prompt optimization via adversarial in-context learning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7308–7327, Bangkok, Thailand. Association for Computational Linguistics.

Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey on in-context learning. arXiv preprint arXiv:2301.00234.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony S. Hartshorn, Aobo Yang, Archi Mitra, Archie Sravanakumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Cantón Ferrer, Cyrus Nikolaidis, Damien Alionsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esibou, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab A. AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriele Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guanglong Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Laurens Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Ju-Qing Jia, Kalyan Vasuntha Alwala, K. Upasani, Kate Plawiak, Keqian Li, Ken-591 neth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline C. Muzzi, Mahesh Babu Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melissa Hall Melanie Kamadbur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri S. Chatterji, Olivier Duchenne, Onur ccelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Lu, Petar Vasic, Peter Weng, Prajwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharar Narang, Sharath Chandra Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gouget, Virginie Do, Vish Vogeti, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenjin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xin-feng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yiqian Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zhengxu Yan, Zhengxing Chen, Zoe Papakipos, Aditya K. Singh, Aaron Grattafiore, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adi Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesen-berg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bhamrabe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Ben Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris
Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Shang-Wen Li, Danny Wyatt, David Adkins, David Xu, Davide Tessugine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ongenel, Francesco Caggioni, Francisco Guzmán, Frank J. Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Bader, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory G. Sizov, Guangyi Zhang, Gunu Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Han Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Asperegn, Hunter Goldman, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gay, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jin Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kaixing(Kai) Wu, U KamHou, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kulal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, A Lavender, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Werhstredt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macev, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Parv Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollár, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghatham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Saradhi Ramaswamy, Shaun Lindsey, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sung-Bae Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Andrei Poenaru, Vlad T. Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xia Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu Wang, Yuchen Hao, Yundi Qian, Yuze He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoqiu Wen, Zhenyu Yang, and Zhiwei Zhao. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Yann Dubois, Percy Liang, and Tatsunori Hashimoto. 2024. Length-controlled alpacaevl: A simple debiasing of automatic evaluators. In First Conference on Language Modeling.

Angela Fan, David Grangier, and Michael Auli. 2018a. Controllable abstractive summarization. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 45–54, Melbourne, Australia. Association for Computational Linguistics.

Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558–3567, Florence, Italy. Association for Computational Linguistics.

Angela Fan, Mike Lewis, and Yann Dauphin. 2018b. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898, Melbourne, Australia. Association for Computational Linguistics.

Marcio Fonseca and Shay B Cohen. 2024. Can large language model summarizers adapt to diverse scientific communication goals? arXiv preprint arXiv:2401.10415.

Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166.

Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. The WebNLG challenge: Generating text from RDF data. In Proceedings of the 10th International Conference on Natural Language Generation, pages 124–133, Santiago de Compostela, Spain. Association for Computational Linguistics.

Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70–79, Hong Kong, China. Association for Computational Linguistics.
Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. 2024. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In The Twelfth International Conference on Learning Representations.

Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. 2023. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints.

Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubashshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4693–4703, Online. Association for Computational Linguistics.

Junxian He, Wojciech Kryscinski, Bryan McCann, Nazneen Rajani, and Caiming Xiong. 2022. CTRL-sum: Towards generic controllable text summarization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5879–5915, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. In International Conference on Learning Representations.

Yukun Huang, Yixin Liu, Raghuveer Thirukovalluru, Arman Cohan, and Bhuwan Dhingra. 2024. Calibrating long-form generations from large language models. arXiv preprint arXiv:2402.06544.

Pegah Jandaghi, XiangHai Sheng, Xinyi Bai, Jay Pujara, and Hakim Sidahmed. 2023. Faithful persona-based conversational dataset generation with large language models. arXiv preprint arXiv:2312.10007.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Klaus Krippendorff. 2022. The reliability of generating data. Chapman and Hall/CRC.

Philippe Laban, Jesse Vig, Wojciech Kryscinski, Shafiq Joty, Caiming Xiong, and Chien-Sheng Wu. 2023. SWiPE: A dataset for document-level simplification of Wikipedia pages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10674–10695, Toronto, Canada. Association for Computational Linguistics.

Surafel Melaku Lakew, Mattia Di Gangi, and Marcello Federico. 2019. Controlling the output length of neural machine translation. In Proceedings of the 16th International Conference on Spoken Language Translation, Hong Kong. Association for Computational Linguistics.

Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2024. Pre-trained language models for text generation: A survey. ACM Computing Surveys, 56(9):1–39.

Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1823–1840, Online. Association for Computational Linguistics.

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

Jianhua Lin. 1991. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 37(1):145–151.

Do Xuan Long, Duong Ngoc Yen, Anh Tuan Luu, Kenji Kawaguchi, Min-Yen Kan, and Nancy F. Chen. 2024. Multi-expert prompting improves reliability, safety and usefulness of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20370–20401, Miami, Florida, USA. Association for Computational Linguistics.

Albert Lu, Hongxin Zhang, Yanzhe Zhang, Xuezhi Wang, and Diyi Yang. 2023. Bounding the capabilities of large language models in open text generation with prompt constraints. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1982–2008, Dubrovnik, Croatia. Association for Computational Linguistics.

Louis Martin, Éric de la Clergerie, Benoît Sagot, and Antoine Bordes. 2020. Controllable sentence simplification. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4689–4698, Marseille, France. European Language Resources Association.

Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FactScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076–12100, Singapore. Association for Computational Linguistics.

OpenAI. 2022. Introducing chatgpt.

OpenAI. 2023. Gpt-4 is openai’s most advanced system, producing safer and more useful responses.
OpenAI. 2024. Hello gpt-4o.

Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lambert Pougouem Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, and Reyhaneh Jabbarvand. 2024. Lost in translation: A study of bugs introduced by large language models while translating code. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–13.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094, Online. Association for Computational Linguistics.

Karl Pearson. 1895. VII. note on regression and inheritance in the case of two parents. proceedings of the royal society of London, 58(347-352):240–242.

Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with “gradient descent” and beam search. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7957–7968, Singapore. Association for Computational Linguistics.

Yevgeniy Puzikov and Iryna Gurevych. 2018. E2E NLG challenge: Neural models vs. templates. In Proceedings of the 11th International Conference on Natural Language Generation, pages 463–471, Tilburg University, The Netherlands. Association for Computational Linguistics.

Chengwei Qin, Aston Zhang, Anirudh Dagar, and Wenming Ye. 2023. In-context learning with iterative demonstration selection. arXiv preprint arXiv:2310.09881.

Qwen Team. 2024. Qwen2.5: A party of foundation models.

Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732.

Jie Ren, Yao Zhao, Tu Vu, Peter J Liu, and Balaji Lakshminarayanan. 2023. Self-evaluation improves selective generation in large language models. In Proceedings on, pages 49–64. PMLR.

Kyle Richardson, Sina Zarrieb, and Jonas Kuhn. 2017. The Code2Text challenge: Text generation in source libraries. In Proceedings of the 10th International Conference on Natural Language Generation, pages 115–119, Santiago de Compostela, Spain. Association for Computational Linguistics.

Nikunj Saunshi, Sadhika Malladi, and Sanjeev Arora. 2020. A mathematical exploration of why language models help solve downstream tasks. arXiv preprint arXiv:2010.03648.

Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073–1083, Vancouver, Canada. Association for Computational Linguistics.

Aaditya K Singh and DJ Strouse. 2024. Tokenization counts: the impact of tokenization on arithmetic in frontier llms. arXiv preprint arXiv:2402.14903.

Jiao Sun, Yufei Tian, Wangchunshu Zhou, Nan Xu, Qian Hu, Rahul Gupta, John Wieting, Nanyun Peng, and Xuezhe Ma. 2023a. Evaluating large language models on controlled generation tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3155–3168, Singapore. Association for Computational Linguistics.

Jiao Sun, Yufei Tian, Wangchunshu Zhou, Nan Xu, Qian Hu, Rahul Gupta, John Frederick Wieting, Nanyun Peng, and Xuezhe Ma. 2023b. Evaluating large language models on controlled generation tasks. arXiv preprint arXiv:2310.14542.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971.

Sara P Wagner. 1963. The abc’s of communication. American Association of Industrial Nurses Journal, 11(8):8–11.

Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023a. Is ChatGPT a good NLG evaluator? a preliminary study. In Proceedings of the 4th New Frontiers in Summarization Workshop, pages 1–11, Singapore. Association for Computational Linguistics.
Rui Wang, Hongru Wang, Fei Mi, Yi Chen, Ruifeng Xu, and Kam-Fai Wong. 2023b. Self-critique prompting with large language models for inductive instructions. arXiv preprint arXiv:2305.13733.

Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. 2024. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. Advances in Neural Information Processing Systems, 36.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022a. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.

Yizhong Wang, Swaroop Mishra, Pegah Alipoormalabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Raysehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Sun. 2022b. Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085–5109, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.

Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2021. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations.

Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2024. Large language models as optimizers. In The Twelfth International Conference on Learning Representations.

Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen, and Wei Cheng. 2023a. Exploring the limits of chatgpt for query or aspect-based text summarization. arXiv preprint arXiv:2302.08081.

Zhao Yang, Yuanzhe Zhang, Dianbo Sui, Cao Liu, Jun Zhao, and Kang Liu. 2023b. Representative demonstration selection for in-context learning with two-stage determinantal point process. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5443–5456, Singapore. Association for Computational Linguistics.

Fan Yin, Jesse Vig, Philippe Laban, Shafiq Joty, Caiming Xiong, and Chien-Sheng Wu. 2023. Did you read the instructions? rethinking the effectiveness of task definitions in instruction learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3063–3079, Toronto, Canada. Association for Computational Linguistics.

Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. BARTScore: Evaluating generated text as text generation. In Advances in Neural Information Processing Systems.

Tianyi Zhang, Varsha Kishore, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitici, Harris Chan, and Jimmy Ba. 2023a. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations.

Yongxin Zhou, Fabien Ringeval, and François Portet. 2023b. Can gpt models follow human summarization guidelines? evaluating chatgpt and gpt-4 for dialogue summarization. arXiv preprint arXiv:2310.16810.
Algorithm 1 LongGuide

Input: \( \mathcal{M} \) and its generation func. \( G_{\mathcal{M}} \), train data \( D^{train} = \{(x_i^t, y_i^t)\}_{i=1}^n \), linguistic processor \( L \).
Input: Task instruction \( I \), instruction to select metrics \( I_M \), score metrics \( I_{score} \), generate MG \( I_{MG} \).
    • Step 1: Metric Collection & Selection
        1: Collect the set of widely-used evaluation metrics \( S \)
        2: \( M = [] \) \( \triangleright \) the set of selected metrics
        3: for \( K \) training iterations do
            4: Sample a batch \( B \) from \( D^{train} \)
            5: \( S_{sub} := G_{\mathcal{M}}([I_M, B, S]) \) \( \triangleright \) top-5 metrics selected from \( S \) for best evaluating \( B \)
            6: \( M = M \cup S_{sub} \)
        end for
        8: \( M = M.sort() \)
    • Step 2: Metric Score Collection via Self-evaluation
        9: \( s_{M_1} = \cdots = s_{M_m} = 0 \) \( \triangleright \) the self-evaluated average scores of selected metrics
        10: for \( i, (x, y) \) in enumerate(\( D^{train} \)) do
            11: \( \{s_{M_1}^i, ..., s_{M_m}^i\} := G_{\mathcal{M}}([I_{score}, x, y, M]) \) \( \triangleright \) self-evaluation
            12: Update \( s_{M_j} = s_{M_j} + (s_{M_j}^i - s_{M_j})/(i+1) \) for all \( j \) in range(m)
        end for
        14: \( score_M = [s_{M_1}, \cdots, s_{M_m}] \)
    • Step 3: Generating Metric Guideline
        15: \( \{d_{M_1}, ..., d_{M_m}\} := G_{\mathcal{M}}([I_{MG}, scores_M, M]) \) \( \triangleright \) generate metrics’ definitions w.r.t scores
        16: MG = joined with newline (\( \{d_{M_1}, ..., d_{M_m}\} \))
    • Step 4: Output Constraint Guideline
        17: Using \( L \) to compute (\( min_s, max_s, avg_s \)) of #sentences in \( y_i^t \), and (\( min_t, max_t, avg_t \)) of #tokens
        18: OCG = “The response must have from {min_s} to {max_s} sentences and from {min_t} to {max_t} words with an average of {avg_t} words and {avg_s} sentences.”
    • Step 5: MG–OCG selection
        19: \( G = \{w/o\ guideline, MG, OCG, MG \& OCG\} \)
        20: \( LongGuide = \arg\max_{g \in G}(per\ formance(\mathcal{M}|I, g, D^{train})) \) \( \triangleright \) automatic guideline selection
Output: LongGuide

A  LongGuide Algorithm

Algorithm 1 outlines the LongGuide algorithm, which generates task-specific guidelines to enhance LLM performance. It selects relevant evaluation metrics, computes self-evaluated scores, formulates metric-based and structural constraints, and automatically chooses the most effective guideline. For detailed step descriptions, see §3.

B  Theoretical Intuitions

We now present a theoretical analysis to explain the observed phenomena.

Task formulation. A long-form generation train dataset for a task \( T \) with \( n \) samples is defined as \( D = \{(x_i^t, y_i^t)\}_{i=1}^n \), where \( x_i^t \) and \( y_i^t \) are the input context and ground-truth sentence- or paragraph-long responses. In the instruction-based setting, an LLM \( \mathcal{M} \) is expected to generate \( y \) given \( x \) and an input instruction \( I \). Recall that \( P_{\mathcal{M}} \) and \( P_T \) are the probability functions of \( \mathcal{M} \)’s generation and the task data distribution, then:

Remark B.1. Under mild assumptions and \( P_{\mathcal{M}} \neq P_T \), there exists \( x \in \mathcal{X} \) such that \( P_{\mathcal{M}}(X = x|D_f) \neq P_T(X = x) \).

The proof is provided in §B.1. In essence, Remark B.1 asserts that when \( \mathcal{M} \) fails to capture the true task data distribution, \( d_f \) cannot recover the desired alignment in the limit. As a result, certain task language and format properties, even when well-presented in demonstrations, may not be implicitly
captured and preserved during LLM generation. We term this unsolved issue the **text property transfer (PT)** challenge: ensuring that \( \mathcal{M} \) captures and preserves specific desired text properties observed in a limited set of labeled data to responses. *We hypothesize that explicitly guiding the model with essential text property criteria mitigates the mismatch identified in Remark B.1 between the induced and task-specific language distributions, leading to improved performance.*

**Definition B.1.** For the task \( T \triangleq \{D, \mathcal{L}\} \), a text property \( T_1 \) of task \( T \) is defined as the task \( T_1 \triangleq \{D_1, \mathcal{L}_1\} \), where \( D_1 = \{(x_i^t, f_1(y_i^t))\}_{i=1}^n \) is the train data of \( T_1 \) mapped from \( D \) by a text property metric (*feature map*) \( f_1 : \mathcal{Y} \to \mathbb{R} \), with \( \mathcal{L} \) and \( \mathcal{L}_1 \) being the objectives of \( T \) and \( T_1 \).

Suppose that the objective is *minimize*\( \theta \in \Theta \mathcal{L}(\theta, T) \) with \( \theta \) being a tunable factor of \( \mathcal{M} \), which can be its parameters or input instruction, and \( \Theta \) being its space. We propose the following hypothesis verified in §4.1:

**Hypothesis B.1.** *T can be approximated by r well-chosen text property tasks \( T_1, ..., T_r \) (\( T \approx T_1 \oplus ... \oplus T_r \)) with corresponding objectives \( \mathcal{L}_1, ..., \mathcal{L}_r \) such that by jointly optimizing them during the generation process, we can approximately optimize T’s objective, i.e., \( \arg\min_{\theta \in \Theta} \mathcal{L} \approx \arg\min_{\theta \in \Theta} \sum_{i=1}^r \mathcal{L}_i \).*

**B.1 Proof of Remark B.1**

**Assumption B.1.** *There exists \( x \in \mathcal{X} \) for which \( P_{\mathcal{M}}(X = x) \neq P_T(X = x) \).*

This assumption is intuitive and realistic, recognizing that LLMs cannot fully capture the vast and nuanced complexity of real-world language beyond their training data. It contradicts the common assumption \( P_{\mathcal{M}}(X) = P_T(X) \) made by prior studies (Xie et al., 2021; Wang et al., 2024). A simple empirical evidence is provided in §C.1. We also assume:

**Assumption B.2.** *Two probability functions are functionally zero equivalent if they act on the same input space and any arbitrary event causes both functions to be zero or non-zero. We assume that \( P_T \) and \( P_{\mathcal{M}} \) are functionally zero equivalent, i.e., \( P_{\mathcal{M}}(X = x) = 0 \Leftrightarrow P_T(X = x) = 0 \ \forall x \in \mathcal{X} \).*

Note that **Assumption B.2** is a relaxed version of the common assumption \( P_{\mathcal{M}}(X) = P_T(X) \), and does not conflict with **Assumption B.1**.

*Proof of Remark B.1.* We prove this theorem by contradiction. Suppose the negation of **Remark B.1** is true, i.e., there exists a \( D_1 \in \mathcal{D} \) such that \( \forall X \in \mathcal{X}, P_{\mathcal{M}}(X|D_1) = P_T(X) \) (*SI*).

Now, let us consider the event \( X \cap D_1^c \) where \( D_1^c \) is the conjugate of event \( D_1 \), or \( D_1^c = \mathcal{D} \setminus D_1 \). We have \( P_{\mathcal{M}}(X \cap D_1^c|D_1) = 0 \). From the assumption (*SI*), we derive \( P_T(X \cap D_1^c) = 0 \). From the **Assumption B.2**, since \( P_{\mathcal{M}} \) and \( P_T \) are functionally zero equivalent, we have \( P_{\mathcal{M}}(X \cap D_1^c) = 0 \).

Similarly, we can consider the event \( X^c \cap D_1^c \) where \( X^c \) is the conjugate of \( X \), we arrive at \( P_{\mathcal{M}}(X^c \cap D_1^c) = 0 \).

Since the two \( X \cap D_1^c \) and \( X^c \cap D_1^c \) form a disjoint union of \( D_1^c \), we derive \( P_{\mathcal{M}}(D_1^c) = P_{\mathcal{M}}(X \cap D_1^c) + P_{\mathcal{M}}(X^c \cap D_1^c) = 0 + 0 = 0 \). Since \( D_1 \) and \( D_1^c \) form a disjoint union of \( \mathcal{D} \), we have \( P_{\mathcal{M}}(D_1) = 1 \).

From the negation statement (*SI*), we have \( P_{\mathcal{M}}(X|D_1) = P_T(X) \ \forall X \in \mathcal{X} \). Since \( X \cap D_1 \) and \( X \cap D_1^c \) form a disjoint union of \( X \), we have \( P_{\mathcal{M}}(X) = P_{\mathcal{M}}(X \cap D_1) + P_{\mathcal{M}}(X \cap D_1^c) = P_{\mathcal{M}}(X \cap D_1) + 0 = P_{\mathcal{M}}(X \cap D_1) \). We also have \( P_{\mathcal{M}}(X|D_1) = \frac{P_{\mathcal{M}}(X \cap D_1)}{P_{\mathcal{M}}(D_1)} \) from Bayes’s theorem, meaning that \( P_{\mathcal{M}}(X|D_1) = P_{\mathcal{M}}(X \cap D_1) = P_{\mathcal{M}}(X) \) (since \( P_{\mathcal{M}}(D_1) = 1 \)). Meanwhile, from the negation statement (*SI*), we have \( P_{\mathcal{M}}(X|D_1) = P_T(X) \), thus \( P_{\mathcal{M}}(X) = P_T(X) \) for all \( X \in \mathcal{X} \), which contradicts to our **Assumption B.1**. Therefore, our negation statement (*SI*) is false, leading to **Remark B.1** is true. \( \square \)

**C Empirical Case Studies Supporting §2**

**C.1 Empirical Case Study for Assumption B.1**

Here, we provide empirical evidence for **Assumption B.1**. It posits that for a given long-form generation test task \( T \), the language model \( \mathcal{M} \) does not entirely encapsulate \( T \)’s true language distribution. This assumption aligns with real-world scenarios since LLMs are not infinitely powerful: there exist task instances where they are out-of-distribution (OOD) for \( \mathcal{M} \).
For example, let us consider Remark B.1 as the task \( T \) and \( \mathcal{M} \) be ChatGPT (OpenAI, 2022). The true language distribution is the specific vocabulary, technical jargon, and nuanced writing style typically found in mathematics. The full dialogue can be found at https://chatgpt.com/share/0189761f-202b-46e6-bab0-39a80bd5340e.

Input: Given two probability functions \( P_1 \) and \( P_2 \). Suppose that there exists at least \( X \) such that \( P_1(X) \neq P_2(X) \). Prove that for any \( D \), there exists at least \( X' \) such that \( P_1(X'|D) \neq P_2(X') \).

ChatGPT output (30 Jul 24): To prove the statement, we need to understand the conditions under which the equality of the probability functions \( P_1 \) and \( P_2 \) given a condition \( D \) breaks down.
Let’s start by stating the given conditions more clearly:
1. We have two probability functions \( P_1 \) and \( P_2 \).
2. There exists at least one event \( X \) such that \( P_1(X) \neq P_2(X) \).
We need to show that for any event \( D \), there exists at least one event \( X' \) such that \( P_1(X'|D) \neq P_2(X'|D) \).
Let’s break it down into steps...

Based on the above example, we can see that ChatGPT does not focus on proving \( P_1(X'|D) \neq P_2(X') \). Instead, it misunderstood as proving \( P_1(X'|D) \neq P_2(X'|D) \), showing that the ChatGPT’s language distribution when solving the task is different from the true task language distribution.

C.2 Empirical Case Study for Remark B.1

Empirical evidence supporting Remark B.1 is not possible to establish since it is an extremal result concerning existence and non-existence. Essentially, Remark B.1 says that if at the beginning, the two distributions of the task and language model are not the same (“first not the same”) but functionally zero equivalent, then for any demonstrations, the two distributions of the task and language model conditioned on those demonstrations are not the same (“second not the same”).

It’s important to note that the data point causing the “first not the same” can differ from the data point causing the “second not the sam”, and this “second not the same” data point needs to be examined by all possible demonstrations. This makes it difficult to empirically verify the theorem since the demonstration space is vast.

D LongGuide’s Extra Preliminary Properties

D.1 LongGuide can Improve Non-instruct Models

Using guidelines learned by LongGuide, we add more instructions to models. Therefore, we aim to examine whether non-instruct models can benefit from these guidelines. Our final conclusion is yes, LongGuide has strong potential to enhance non-instruct models.

Setups. Since non-instruct models might struggle to follow our instructions to generate the guidelines §7, we utilize the guidelines learned by an instruct model instead. We run our experiments with Mistral-7B-v0.14(Jiang et al., 2023) using the guidelines learned by Mistral-7B-Instruct-v0.2.

Findings. The results are provided in Table 7. We observe that LongGuide improves more than half of the experiments, showing its potential effectiveness in enhancing even non-instruct models, especially for the translation task.

<table>
  <tr>
    <th>Methods</th>
    <th>CNN (3.0.0)</th>
    <th>IWSLT17</th>
    <th>CommGen-Chall.</th>
  </tr>
  <tr>
    <td>Zero-shot (ZS)</td>
    <td>7.60<sub>±0.58</sub></td>
    <td>2.99<sub>±0.83</sub></td>
    <td>10.96<sub>±0.36</sub></td>
  </tr>
  <tr>
    <td>+ OCG<sup>†</sup></td>
    <td>6.60<sub>±0.74</sub></td>
    <td>3.70<sub>±0.29</sub></td>
    <td>10.72<sub>±0.56</sub></td>
  </tr>
  <tr>
    <td>+ MG</td>
    <td><b>9.04<sub>±1.02</sub>↑</b></td>
    <td><b>5.39<sub>±0.93</sub>↑</b></td>
    <td>8.55<sub>±0.74</sub></td>
  </tr>
  <tr>
    <td>+ MG-OCG</td>
    <td>8.38<sub>±0.91</sub></td>
    <td>4.59<sub>±0.97</sub></td>
    <td>7.99<sub>±0.76</sub></td>
  </tr>
  <tr>
    <td>+ LongGuide</td>
    <td><b>9.04<sub>±1.02</sub>↑</b></td>
    <td><b>5.39<sub>±0.93</sub>↑</b></td>
    <td><b>10.96<sub>±0.36</sub></b></td>
  </tr>
  <tr>
    <td>Few-shot (FS)</td>
    <td>3.14<sub>±0.32</sub></td>
    <td>3.44<sub>±0.83</sub></td>
    <td>4.67<sub>±0.43</sub></td>
  </tr>
  <tr>
    <td>+ OCG</td>
    <td>2.34<sub>±0.21</sub></td>
    <td>3.56<sub>±0.61</sub></td>
    <td>8.11<sub>±0.63</sub></td>
  </tr>
  <tr>
    <td>+ MG</td>
    <td><b>3.24<sub>±0.26</sub></b></td>
    <td>6.65<sub>±0.97</sub></td>
    <td><b>10.71<sub>±0.80</sub></b></td>
  </tr>
  <tr>
    <td>+ MG-OCG</td>
    <td>2.99<sub>±0.29</sub></td>
    <td><b>7.88<sub>±0.91</sub></b></td>
    <td>9.39<sub>±0.89</sub></td>
  </tr>
  <tr>
    <td>+ LongGuide</td>
    <td>2.24<sub>±0.21</sub></td>
    <td><b>7.88<sub>±0.91</sub></b></td>
    <td><b>10.71<sub>±0.80</sub></b></td>
  </tr>
</table>

Table 7: ROUGE-L performance of Mistral-7B-v0.1 using LongGuide learned by Mistral-7B-Instruct-v0.2. We observe that LongGuide improves more than half of the experiments, showing its potential effectiveness in enhancing even non-instruct models, especially for the translation task.

4https://huggingface.co/mistralai/Mistral-7B-v0.1
D.2 LongGuide can be Transferable from Weaker to Stronger Models

We find that the guidelines learned by LongGuide are transferable from weaker to stronger models. A weaker model can learn the guidelines at a low cost, which can then be used to enhance the performance of stronger models. This is particularly advantageous because powerful models are often closed-source and expensive to query, whereas open-source models are weaker but free to use.

Setups. We demonstrate this through experiments on CNN (3.0.0), IWSLT17 en-ja, and CommGen-Chall, representing all the tasks. We used the MG generated by Mistral for experiments on ChatGPT and vice versa under both zero-shot and few-shot settings.

Findings. Table 8 outlines the results. We observe that Mistral’s MG generally improves ChatGPT performance, but not vice versa. Explaining these phenomena, firstly, the OCG is transferable across models because it is independent of any specific model. Secondly, the MG, while it helps models capture task distributions, an MG learned from a stronger model may not benefit a weaker model, as the weaker model may misinterpret it. In contrast, the stronger model, with better text comprehension, can generalize task distributions from MG even when MG is poor and/or not well expressive generated by the weaker model.

<table>
  <tr>
    <th>Methods</th>
    <th>CNN (3.0.0)</th>
    <th>IWSLT17 en-ja</th>
    <th>CommGen-Chall.</th>
  </tr>
  <tr>
    <td>ChatGPT Zero-shot (ZS)</td>
    <td>20.12±0.27</td>
    <td>36.13±0.87</td>
    <td>24.21±0.37</td>
  </tr>
  <tr>
    <td>ChatGPT ZS w/ Mistral’s MG</td>
    <td>21.41±0.62↑</td>
    <td>39.66±1.47↑</td>
    <td>29.95±2.06↑</td>
  </tr>
  <tr>
    <td>ChatGPT Few-shot (FS)</td>
    <td>14.51±0.80</td>
    <td>31.93±1.88</td>
    <td>22.08±0.63</td>
  </tr>
  <tr>
    <td>ChatGPT FS w/ Mistral’s MG</td>
    <td>13.96±1.50↑</td>
    <td>32.34±1.79↑</td>
    <td>33.34±1.56↑</td>
  </tr>
  <tr>
    <td>Mistral Zero-shot (ZS)</td>
    <td>19.23±0.34</td>
    <td>13.12±1.39</td>
    <td>10.12±0.02</td>
  </tr>
  <tr>
    <td>Mistral w/ ChatGPT’s MG</td>
    <td>19.67±0.71↑</td>
    <td>7.98±1.49↓</td>
    <td>6.29±1.06↓</td>
  </tr>
  <tr>
    <td>Mistral Few-shot (FS)</td>
    <td>17.56±0.63</td>
    <td>12.69±1.82</td>
    <td>3.89±0.17</td>
  </tr>
  <tr>
    <td>Mistral FS w/ ChatGPT’s MG</td>
    <td>19.00±1.92↑</td>
    <td>11.86±2.79↓</td>
    <td>3.61±0.38↓</td>
  </tr>
</table>

Table 8: LongGuide can be transferable from weaker to stronger models, evaluated by ROUGE-L.

D.3 LongGuide can be Compared and Combined with Automatic Prompt Optimization Algorithms

The MG and OCG learned by LongGuide may not be fully optimized for LLMs. Hence, it’s intuitive to suggest that LLMs could achieve even greater performance by adopting optimal guidelines. In this section, we illustrate that the guidelines learned by LongGuide can be further refined through discrete prompt optimization algorithms. This capability is advantageous for LongGuide, enabling its concurrent development and integration with automatic prompt optimization algorithms.

Setup. We employ two strong prompt optimizers, APO (Pryzant et al., 2023) and adv-ICL (Do et al., 2024), in our experiments. We also compare LongGuide with EvolPrompt (Guo et al., 2024) in this section. Here is our methodology: we integrated the guidelines generated by LongGuide into the prompt, including the input instruction and demonstrations. Subsequently, we applied the prompt optimizers to refine the input instruction, demonstrations, and guidelines. Our experiments were conducted using Mistral on datasets including CNN, IWSLT 2017 en-ja, and CommonGen-Challenge. Following our findings in Table 4, the guideline being optimized for CNN and IWSLT 2017 en-ja is OCG, while for CommonGen-Challenge is MG-OCG.

Findings. Our results are detailed in Table 9. In summary, when further optimizing the OCG using APO and adv-ICL for CNN and IWSLT 2017, we observed a slight improvement. This could be attributed to the OCG already being concise and straightforward, making it easier for models to grasp. However, for the CommonGen-Challenge dataset, which utilizes the

<table>
  <tr>
    <th>Methods</th>
    <th>CNN (3.0.0)</th>
    <th>IWSLT17</th>
    <th>CommGen-Chall.</th>
  </tr>
  <tr>
    <td>Zero-shot (ZS)</td>
    <td>19.23±0.34</td>
    <td>13.12±1.39</td>
    <td>10.12±0.02</td>
  </tr>
  <tr>
    <td>+ APO</td>
    <td>19.53±0.28</td>
    <td>14.45±1.84</td>
    <td>11.21±2.02</td>
  </tr>
  <tr>
    <td>+ EvolPrompt</td>
    <td>20.16±3.44</td>
    <td>15.04±2.12</td>
    <td>14.06±3.02</td>
  </tr>
  <tr>
    <td>+ adv-ICL</td>
    <td>18.87±2.69</td>
    <td>15.01±1.72</td>
    <td>13.12±2.21</td>
  </tr>
  <tr>
    <td>+ LongGuide</td>
    <td>22.46±0.64</td>
    <td>16.53±0.59</td>
    <td>25.20±1.89</td>
  </tr>
  <tr>
    <td>+ LongGuide + APO</td>
    <td><b>22.76±1.06↑</b></td>
    <td><b>17.13±1.05↑</b></td>
    <td><b>27.01±1.01↑</b></td>
  </tr>
  <tr>
    <td>+ LongGuide + adv-ICL</td>
    <td><b>21.97±3.21↓</b></td>
    <td><b>16.90±2.15↑</b></td>
    <td><b>26.18±3.47↑</b></td>
  </tr>
</table>

Table 9: Guidelines learned by LongGuide are further optimized by discrete prompt optimization frameworks bringing even better performance, with Mistral, evaluated by ROUGE-L.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">Sum.</th>
    <th colspan="3">Simplification</th>
    <th colspan="3">Translation</th>
    <th colspan="3">Dialogue Gen.</th>
    <th colspan="3">Table2Text</th>
  </tr>
  <tr>
    <th>SAMSum</th>
    <th>CNN (3.0)</th>
    <th>XL-Sum</th>
    <th>SWiPE</th>
    <th>IWSLT17 en-ja</th>
    <th>Syn. Persona</th>
    <th>Comm.-Chall.</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>#shots (ran.)</td>
    <td>3</td>
    <td>3</td>
    <td>5</td>
    <td>3</td>
    <td>3</td>
    <td>5</td>
    <td>5</td>
    <td>5</td>
  </tr>
  <tr>
    <td>Zero-shot (ZS)</td>
    <td>22.20 / 20.05</td>
    <td>19.23 / 20.43</td>
    <td>9.19 / 8.82</td>
    <td>36.60 / 39.01</td>
    <td>13.12 / 13.72</td>
    <td>12.76 / 11.79</td>
    <td>10.12 / 6.19</td>
    <td>17.38</td>
  </tr>
  <tr>
    <td>+ APO</td>
    <td>23.77 / 22.02</td>
    <td>19.53 / 21.46</td>
    <td>12.06 / 11.50</td>
    <td>36.92 / 39.41</td>
    <td>14.45 / 15.49</td>
    <td>10.66 / 10.05</td>
    <td>11.21 / 7.12</td>
    <td>18.26</td>
  </tr>
  <tr>
    <td>+ <b>LongGuide</b></td>
    <td><b>28.35 / 28.79</b></td>
    <td><b>22.46 / 27.82</b></td>
    <td><b>14.38 / 14.13</b></td>
    <td><b>38.21 / 40.83</b></td>
    <td><b>16.53 / 18.81</b></td>
    <td><b>14.69 / 12.86</b></td>
    <td><b>25.20 / 24.03</b></td>
    <td><b>23.37</b></td>
  </tr>
  <tr>
    <td>% gain (+)</td>
    <td>6.15 / 8.74</td>
    <td>3.23 / 7.39</td>
    <td>5.19 / 3.31</td>
    <td>1.61 / 1.82</td>
    <td>3.41 / 5.09</td>
    <td>1.93 / 1.07</td>
    <td>15.08 / 17.84</td>
    <td>5.99</td>
  </tr>
  <tr>
    <td>Few-shot (FS)</td>
    <td>27.13 / 27.21</td>
    <td>17.56 / 20.55</td>
    <td>9.79 / 8.32</td>
    <td>39.47 / 39.76</td>
    <td>12.69 / 13.78</td>
    <td>3.56 / 2.67</td>
    <td>3.98 / 1.94</td>
    <td>16.32</td>
  </tr>
  <tr>
    <td>+ APO</td>
    <td>26.23 / 25.88</td>
    <td>18.18 / 21.32</td>
    <td>11.99 / 11.71</td>
    <td>39.55 / 39.56</td>
    <td>14.08 / 14.70</td>
    <td>4.26 / 2.91</td>
    <td>5.45 / 3.76</td>
    <td>17.12</td>
  </tr>
  <tr>
    <td>+ <b>LongGuide</b></td>
    <td><b>30.65 / 31.72</b></td>
    <td><b>19.19 / 22.30</b></td>
    <td><b>15.23 / 14.02</b></td>
    <td><b>41.36 / 41.22</b></td>
    <td><b>16.62 / 17.92</b></td>
    <td><b>5.25 / 4.46</b></td>
    <td><b>25.05 / 21.90</b></td>
    <td><b>21.92</b></td>
  </tr>
  <tr>
    <td>% gain (+)</td>
    <td>3.52 / 4.51</td>
    <td>1.63 / 1.75</td>
    <td>5.44 / 3.70</td>
    <td>1.89 / 1.46</td>
    <td>3.66 / 4.14</td>
    <td>1.69 / 1.79</td>
    <td>21.07 / 19.96</td>
    <td>5.61</td>
  </tr>
  <tr>
    <td>Zero-shot (ZS)</td>
    <td>23.83 / 20.23</td>
    <td>20.12 / 24.11</td>
    <td>10.80 / 11.46</td>
    <td>45.09 / 43.28</td>
    <td>36.13 / 38.32</td>
    <td>19.46 / 19.75</td>
    <td>24.21 / 24.04</td>
    <td>25.77</td>
  </tr>
  <tr>
    <td>+ APO</td>
    <td>25.05 / 22.99</td>
    <td>20.34 / 21.88</td>
    <td>12.19 / 12.52</td>
    <td><b>46.32 / 44.89</b></td>
    <td>37.74 / 39.01</td>
    <td>19.91 / 19.80</td>
    <td>23.63 / 24.18</td>
    <td>26.45</td>
  </tr>
  <tr>
    <td>+ <b>LongGuide</b></td>
    <td><b>30.47 / 28.35</b></td>
    <td><b>22.19 / 30.79</b></td>
    <td><b>20.93 / 22.61</b></td>
    <td>45.09 / 43.28</td>
    <td><b>41.27 / 43.79</b></td>
    <td><b>22.98 / 23.79</b></td>
    <td><b>34.41 / 36.84</b></td>
    <td><b>31.97</b></td>
  </tr>
  <tr>
    <td>% gain (+)</td>
    <td>6.64 / 8.14</td>
    <td>2.07 / 6.68</td>
    <td>10.13 / 11.15</td>
    <td>0.00 / 0.00</td>
    <td>5.09 / 5.47</td>
    <td>3.52 / 4.04</td>
    <td>10.20 / 12.80</td>
    <td>6.13</td>
  </tr>
  <tr>
    <td>Few-shot (FS)</td>
    <td>22.21 / 25.37</td>
    <td>14.51 / 17.52</td>
    <td>11.42 / 10.83</td>
    <td>33.72 / 32.69</td>
    <td>31.93 / 32.68</td>
    <td>16.10 / 18.10</td>
    <td>22.08 / 23.52</td>
    <td>22.34</td>
  </tr>
  <tr>
    <td>+ APO</td>
    <td>24.22 / 22.77</td>
    <td>15.20 / 17.04</td>
    <td>14.07 / 15.69</td>
    <td>34.46 / 33.18</td>
    <td>33.72 / 35.50</td>
    <td>17.68 / 17.77</td>
    <td>25.09 / 24.70</td>
    <td>23.65</td>
  </tr>
  <tr>
    <td>+ <b>LongGuide</b></td>
    <td><b>31.46 / 30.04</b></td>
    <td><b>18.17 / 17.82</b></td>
    <td><b>19.95 / 22.49</b></td>
    <td><b>37.60 / 35.66</b></td>
    <td><b>38.43 / 42.84</b></td>
    <td><b>22.36 / 20.31</b></td>
    <td><b>38.21 / 37.64</b></td>
    <td><b>29.35</b></td>
  </tr>
  <tr>
    <td>% gain (+)</td>
    <td>9.25 / 4.67</td>
    <td>3.66 / 1.00</td>
    <td>5.83 / 11.06</td>
    <td>3.88 / 2.97</td>
    <td>6.50 / 10.16</td>
    <td>6.53 / 2.21</td>
    <td>16.13 / 14.12</td>
    <td>7.21</td>
  </tr>
</table>

Table 10: Supplementary ROUGE-L / BLEU-1 results on seven long-form generation tasks showing that the trends of ROUGE-L and BLEU-1 scores are nearly identical.

MG-OCG guideline with more detail, APO and adv-ICL have a greater amount of material to optimize within the prompts. This led to a substantial improvement in performance compared to the other datasets.

E Supplementary Results and Discussions

E.1 Additional Baselines: Using more Shots for ICL

We supplement the results for CNN (3.0.0), SWiPE, and Comm.-Chall. in Table 11 where we use 10 shots for CNN, 50 shots for SWiPE, and Comm.-Chall up to the window size limit of gpt-3.5-turbo-1106 evaluated by ROUGE-L / GPT-4o-Judge scores.

We observe that while supplementing more shots to ChatGPT improves model’s performance, LongGuide further boosts the ICL performance significantly for all three benchmarks.

<table>
  <tr>
    <th>#shot</th>
    <th>CNN (3.0)</th>
    <th>SWiPE</th>
    <th>Comm.-Chall.</th>
  </tr>
  <tr>
    <td>3-5 shots</td>
    <td>14.51 / 4.38</td>
    <td>33.72 / 5.07</td>
    <td>22.08 / 4.19</td>
  </tr>
  <tr>
    <td>+ LongGuide</td>
    <td><b>18.17 / 4.42</b></td>
    <td><b>37.60 / 5.25</b></td>
    <td><b>38.21 / 7.21</b></td>
  </tr>
  <tr>
    <td>10-50 shots</td>
    <td>20.55 / 6.67</td>
    <td>44.04 / 6.07</td>
    <td>28.18 / 4.85</td>
  </tr>
  <tr>
    <td>+ LongGuide</td>
    <td><b>21.69 / 6.82</b></td>
    <td><b>46.17 / 6.67</b></td>
    <td><b>42.55 / 7.72</b></td>
  </tr>
</table>

Table 11: Performance comparison of models with and without LongGuide across different datasets and shot settings.

To analyze LongGuide’s impact on LLMs, we perform a simple attention analysis to investigate (1) how LLMs attend to LongGuide and (2) utilize the input context when conditioning on LongGuide. Specifically, for (1), we calculate the average attention scores across all heads and layers for each guideline token. For (2), we evaluate the entropy of the attention scores overall context tokens. We experiment with Mistral on 100 SAMSum random samples. We learn two key findings.

Firstly, Mistral shows substantial attention to the guidelines. By using MG, 37.81% of attention is on guideline tokens. For OCG, it is 22.56%, and MG-OCG, 37.87%. Notably, the average attention on OCG tokens is higher than on context, while MG and MG-OCG receive a fair amount, confirming mode attention on guidelines (Appx.-Table 16).

Secondly, from Figure 6, Mistral exhibits more selective context attention when conditioned on guidelines. The largest entropy gap occurs in the first layer, where with guidelines, the model sparsely processes the context but, without them, is biased towards focusing narrowly on specific context parts. In

![Entropy of attention over the input context across 32 Mistral layers.](page_1092_1342_324_246.png)

Figure 6: Entropy of attention over the input context across 32 Mistral layers.
Figure 7: Property maintenance experiments with ICL. See Appx.-Figure 19 for a full example.

Figure 8: Property maintenance experiments with ICL full results. IT is the adding a simple guideline baseline.

the final layer, the model distributes attention more evenly with guidelines than without. Generally, MG stabilizes context use across layers, while OCG shows greater variance, likely because it does not directly control generation quality, therefore, the model bias almost exists as origin, as we can see the trends of using OCG and no guidelines are relatively similar. These findings indicate that guidelines potentially improve context utilization and mitigate token bias.

E.3 LongGuide on Reasoning Tasks

We conduct experiments comparing LongGuide to various baselines on reasoning tasks. We select Mistral as our LLM, and GSM8K (Cobbe et al., 2021) and SVAMP (Patel et al., 2021) as benchmarks for evaluation. For each benchmark, we randomly sampled 200 instances from the test set for assessment and 50 instances from the train set to train the prompt optimizers and LongGuide.

The results are averaged over three runs, and outlined in Table 12. LongGuide slightly outperforms the Zero-shot and Few-shot baselines but falls short compared to prompt optimizers. Nonetheless, the findings confirm that additional instructions for LLMs can potentially improve the init model, leading to further enhanced reasoning performance with prompt optimization.

<table>
  <tr>
    <th>Methods</th>
    <th>GSM8k</th>
    <th>SVAMP</th>
  </tr>
  <tr>
    <td>Zero-shot (ZS)</td>
    <td>39.66</td>
    <td>60.33</td>
  </tr>
  <tr>
    <td>+ APO</td>
    <td>41.83</td>
    <td>62.33</td>
  </tr>
  <tr>
    <td>+ adv-ICL</td>
    <td>42.66</td>
    <td>62.83</td>
  </tr>
  <tr>
    <td>+ LongGuide</td>
    <td>40.83</td>
    <td>63.33</td>
  </tr>
  <tr>
    <td>Few-shot (FS)</td>
    <td>32.33</td>
    <td>61.66</td>
  </tr>
  <tr>
    <td>+ APO</td>
    <td>34.33</td>
    <td>63.00</td>
  </tr>
  <tr>
    <td>+ adv-ICL</td>
    <td>35.00</td>
    <td>62.66</td>
  </tr>
  <tr>
    <td>+ LongGuide</td>
    <td>34.83</td>
    <td>62.83</td>
  </tr>
</table>

Table 12: Performance of LongGuide with Mistral on reasoning tasks.
Figure 9: Pairwise Pearson correlation coefficient of metrics.

<table>
  <tr>
    <th>Source</th>
    <th>Metrics</th>
    <th>#</th>
  </tr>
  <tr>
    <td>The ABC’s of Communication (Wagner, 1963)</td>
    <td>Accuracy, Brevity, Clarity</td>
    <td>3</td>
  </tr>
  <tr>
    <td>BARTScore (Yuan et al., 2021)</td>
    <td>Relevance, Coherence</td>
    <td>2</td>
  </tr>
  <tr>
    <td>GPTScore (Fu et al., 2023)</td>
    <td>Semantic Coverage, Factuality, Fluency, Informativeness, Consistency, Engagement, Specificity, Correctness, Understandability, Diversity</td>
    <td>10</td>
  </tr>
  <tr>
    <td>We propose</td>
    <td>Completeness, Conciseness, Neutrality, Naturalness, Readability, Creativity, Rationalness, Truthfulness, Respect of Chronology, Non-repetitiveness, Indicativeness, Resolution</td>
    <td>12</td>
  </tr>
  <tr>
    <td>Total</td>
    <td></td>
    <td>27</td>
  </tr>
</table>

Table 13: Metrics collected for LongGuide’s metric guideline (MG).

E.4 Supplementary Results for §2

E.5 Understanding MG and OCG: How Do They Work (Together)?

Metric Guideline (MG) (Step 1-3). To understand how models select and evaluate metrics, we analyze the specific metrics chosen for each task, their selection frequencies, and their average scores (Appx.-Table 17 and figs. 12 and 13 respectively). Overall, each of the 27 metrics is selected and evaluated in at least one task. Among them, common linguistic metrics such as “Clarity” are frequently selected, while task-specific metrics like “Creativity” are less frequently chosen. By examining the scores of selected metrics, we find that common linguistic metrics generally achieve high scores, as anticipated. However, task-specific metrics like “Creativity” exhibit varying scores across tasks, indicating their differing importance and relevance. Additionally, we also find that within MG can conflict with each other, such as “Conciseness” and “Informativeness” (see Appx.-Figure 17 for an example). This underscores the importance of LongGuide’s Step 2 in weighting the metrics.

Output Constraint Guideline (OCG) (Step 4). We find that both the token and sentence constraints are crucial for LLMs (Appx.-E.13), with the sentence being more beneficial. We hypothesize that LLMs have better control over the number of sentences than tokens, as counting sentences is intuitively simpler than tokens. This can be observed in our experiment in §2.

MG and OCG are complementary and non-interchangeable. MG and OCG complement each other rather than conflict, as partially discussed in §4.1. This is because MG language metrics primarily concern the characteristics of responses rather than their structural aspects such as sentence and token count, which is the main focus of the OCG. In addition, the MG and OCG are not interchangeable. One might question whether adopting conciseness and brevity metrics could sufficiently alter the OCG, or if the OCG could effectively encompass the MG guideline. Our answer is no. While MG can steer LLMs towards brevity in responses, it lacks precise quantification for conciseness. Modern LLMs, often trained to generate verbose responses, may struggle to meet human conciseness without explicit statistics. Meanwhile, the OCG supplies them in the form of bins and means, yet these statistics alone do not directly address linguistic qualities. We provide examples as evidence supporting our arguments in Appx.-Figures 15 and 16.

E.6 Collected Metrics in LongGuide’s Step 1 (§3)

Table 13 presents our 27 metrics collected for LongGuide’s Step 1.
E.7 JS Divergence over all LongGuide Metrics with SAMSum (§4.1)

![Density plots of MG and OCG metrics selected by Mistral under the few-shot (FS) setting, measured on ground-truth, FS, and FS w/ LongGuide answers. For Jensen–Shannon divergence, lower is better.](page_120_120_1248_393.png)

Figure 10: Density plots of MG and OCG metrics selected by Mistral under the few-shot (FS) setting, measured on ground-truth, FS, and FS w/ LongGuide answers. For Jensen–Shannon divergence, lower is better.

Figure 10 presents density plots of MG and OCG metrics selected by Mistral under the few-shot (FS) setting, measured on ground-truth, FS, and FS w/ LongGuide answers. For Jensen–Shannon divergence, the lower is better.

E.8 Step 5 CD-MG Selection Results of LongGuide (§4.1)

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">Summarization</th>
    <th colspan="3">Simplification</th>
    <th colspan="3">Translation</th>
    <th colspan="3">Dialogue Generation</th>
    <th colspan="3">Table2Text</th>
  </tr>
  <tr>
    <th>SAMSum</th>
    <th>CNN (3.0.0)</th>
    <th>XL-Sum</th>
    <th>SWiPE</th>
    <th>IWSLT17 en-ja</th>
    <th>Synthetic Persona</th>
    <th>CommGen-Chall.</th>
  </tr>
  <tr>
    <td>#shots (random)</td>
    <td>3</td>
    <td>3</td>
    <td>5</td>
    <td>3</td>
    <td>5</td>
    <td>5</td>
    <td>5</td>
    <td>5</td>
  </tr>
  <tr>
    <td>Zero-shot (ZS)</td>
    <td>21.25</td>
    <td>18.96</td>
    <td>8.88</td>
    <td>36.21</td>
    <td>14.05</td>
    <td>12.93</td>
    <td>9.12</td>
  </tr>
  <tr>
    <td>+ OCG</td>
    <td>27.43</td>
    <td><b>21.92</b></td>
    <td><b>14.22</b></td>
    <td>31.19</td>
    <td><b>16.93</b></td>
    <td>12.99</td>
    <td>20.67</td>
  </tr>
  <tr>
    <td>+ MG</td>
    <td>27.68</td>
    <td>18.02</td>
    <td>10.26</td>
    <td><b>36.74</b></td>
    <td>11.06</td>
    <td>13.74</td>
    <td>19.98</td>
  </tr>
  <tr>
    <td>+ MG-OCG</td>
    <td><b>28.34</b></td>
    <td>18.63</td>
    <td>13.90</td>
    <td>35.12</td>
    <td>15.46</td>
    <td><b>14.14</b></td>
    <td><b>20.87</b></td>
  </tr>
  <tr>
    <td>MG-OCG Sel.</td>
    <td>MG-OCG</td>
    <td>OCG</td>
    <td>OCG</td>
    <td>MG</td>
    <td>OCG</td>
    <td>MG-OCG</td>
    <td>MG-OCG</td>
  </tr>
  <tr>
    <td>Mistral</td>
    <td>25.55</td>
    <td>17.30</td>
    <td>9.85</td>
    <td>39.29</td>
    <td>13.52</td>
    <td>6.19</td>
    <td>4.01</td>
  </tr>
  <tr>
    <td>+ OCG</td>
    <td>27.31</td>
    <td>16.45</td>
    <td>12.47</td>
    <td>29.85</td>
    <td><b>17.58</b></td>
    <td>6.45</td>
    <td><b>20.50</b></td>
  </tr>
  <tr>
    <td>+ MG</td>
    <td>27.88</td>
    <td>18.47</td>
    <td>12.01</td>
    <td><b>41.07</b></td>
    <td>14.09</td>
    <td>6.47</td>
    <td>11.16</td>
  </tr>
  <tr>
    <td>+ MG-OCG</td>
    <td><b>30.81</b></td>
    <td><b>18.87</b></td>
    <td><b>14.89</b></td>
    <td>39.80</td>
    <td>17.02</td>
    <td><b>8.06</b></td>
    <td>5.18</td>
  </tr>
  <tr>
    <td>MG-OCG Sel.</td>
    <td>MG-OCG</td>
    <td>MG-OCG</td>
    <td>MG-OCG</td>
    <td>MG</td>
    <td>OCG</td>
    <td>MG-OCG</td>
    <td>OCG</td>
  </tr>
  <tr>
    <td>ChatGPT</td>
    <td>24.21</td>
    <td>19.54</td>
    <td>10.78</td>
    <td><b>45.11</b></td>
    <td>36.22</td>
    <td>19.68</td>
    <td>24.23</td>
  </tr>
  <tr>
    <td>+ OCG</td>
    <td>28.81</td>
    <td>21.88</td>
    <td><b>20.66</b></td>
    <td>37.58</td>
    <td>38.45</td>
    <td><b>23.09</b></td>
    <td><b>35.04</b></td>
  </tr>
  <tr>
    <td>+ MG</td>
    <td>25.12</td>
    <td>20.02</td>
    <td>10.42</td>
    <td>45.09</td>
    <td>37.72</td>
    <td>19.81</td>
    <td>18.50</td>
  </tr>
  <tr>
    <td>+ MG-OCG</td>
    <td><b>29.79</b></td>
    <td><b>21.99</b></td>
    <td><b>19.91</b></td>
    <td>42.72</td>
    <td><b>41.50</b></td>
    <td>20.82</td>
    <td>30.09</td>
  </tr>
  <tr>
    <td>MG-OCG Sel.</td>
    <td>MG-OCG</td>
    <td>MG-OCG</td>
    <td>MG-OCG</td>
    <td>MG-OCG</td>
    <td>MG-OCG</td>
    <td>MG-OCG</td>
    <td>OCG</td>
  </tr>
  <tr>
    <td>Few-shot (FS)</td>
    <td>27.44</td>
    <td>13.77</td>
    <td>12.11</td>
    <td>33.30</td>
    <td>28.76</td>
    <td>17.12</td>
    <td>24.12</td>
  </tr>
  <tr>
    <td>+ OCG</td>
    <td>29.98</td>
    <td><b>17.55</b></td>
    <td><b>19.26</b></td>
    <td>16.22</td>
    <td>35.73</td>
    <td><b>21.50</b></td>
    <td>36.51</td>
  </tr>
  <tr>
    <td>+ MG</td>
    <td>28.89</td>
    <td>14.03</td>
    <td>12.75</td>
    <td>19.14</td>
    <td>36.09</td>
    <td>19.12</td>
    <td>21.99</td>
  </tr>
  <tr>
    <td>+ MG-OCG</td>
    <td><b>30.65</b></td>
    <td>13.12</td>
    <td>18.64</td>
    <td><b>37.24</b></td>
    <td><b>36.22</b></td>
    <td>18.99</td>
    <td><b>36.33</b></td>
  </tr>
  <tr>
    <td>MG-OCG Sel.</td>
    <td>MG-OCG</td>
    <td>OCG</td>
    <td>OCG</td>
    <td>MG-OCG</td>
    <td>MG-OCG</td>
    <td>OCG</td>
    <td>MG-OCG</td>
  </tr>
</table>

Table 14: MG-OCG selection results on \( D^{train} \) set for the main experiments in Table 3, evaluated by ROUGE-L.

The numerical MG-OCG selection results on \( D^{train} \) are presented in Table 14, as also noted in Table 4. Overall, the performance of LongGuide on \( D^{train} \) closely mirrors its performance on the testing tasks in Table 4. The only discrepancy is for the IWSLT17 en-ja task with ChatGPT using few-shot prompting: the optimal guideline combination on \( D^{train} \) is MG-OCG (see Table 14), whereas the best on the testing set is MG (see Table 4).

E.9 LongGuide can Generalize from Demonstrations (§5.2)

Table 15 presents the numerical results of Figure 5 in §5.2. Even with only 3-5 exemplars as demonstrations, LongGuide effectively derives MG and OCG guidelines, benefiting the model. In this case, \( D^{train} \) is the set of demonstrations, and the rest of LongGuide’s steps remain unchanged.
<table>
  <tr>
    <th>Methods</th>
    <th>CNN (3.0.0)</th>
    <th>IWSLT17 en-ja</th>
    <th>CommGen-Chall.</th>
  </tr>
  <tr>
    <td>Zero-shot (ZS)</td>
    <td>19.23\pm0.34</td>
    <td>13.12\pm1.39</td>
    <td>10.12\pm0.02</td>
  </tr>
  <tr>
    <td>+ OCG trained on \( D^{train} \)</td>
    <td><b>22.46\pm0.64</b></td>
    <td>16.53\pm0.59</td>
    <td>24.16\pm0.11</td>
  </tr>
  <tr>
    <td>+ MG trained on \( D^{train} \)</td>
    <td>18.35\pm0.60</td>
    <td>8.71\pm0.53</td>
    <td>21.54\pm7.50</td>
  </tr>
  <tr>
    <td>+ MG-OCG trained on \( D^{train} \)</td>
    <td>22.05\pm0.84</td>
    <td>15.76\pm1.85</td>
    <td><b>25.20\pm1.89</b></td>
  </tr>
  <tr>
    <td>+ LongGuide trained on \( D^{train} \)</td>
    <td><b>22.46\pm0.64</b></td>
    <td>16.53\pm0.59</td>
    <td><b>25.20\pm1.89</b></td>
  </tr>
  <tr>
    <td>- + OCG trained on Demos</td>
    <td>20.46\pm0.10</td>
    <td><b>17.27\pm1.83</b></td>
    <td>23.97\pm0.47</td>
  </tr>
  <tr>
    <td>+ MG trained on Demos</td>
    <td>18.33\pm0.25</td>
    <td>8.63\pm1.08</td>
    <td>18.98\pm0.52</td>
  </tr>
  <tr>
    <td>+ MG-OCG trained on Demos</td>
    <td>19.16\pm0.37</td>
    <td>14.00\pm3.42</td>
    <td>24.46\pm2.43</td>
  </tr>
  <tr>
    <td>+ LongGuide trained on Demos</td>
    <td>20.46\pm0.10</td>
    <td>14.00\pm2.42</td>
    <td>24.46\pm2.43</td>
  </tr>
  <tr>
    <td>Few-shot (FS)</td>
    <td>17.56\pm0.63</td>
    <td>12.69\pm1.82</td>
    <td>3.98\pm0.17</td>
  </tr>
  <tr>
    <td>+ OCG trained on \( D^{train} \)</td>
    <td>19.17\pm1.27</td>
    <td><b>19.86\pm2.93</b></td>
    <td>25.05\pm0.76</td>
  </tr>
  <tr>
    <td>+ MG trained on \( D^{train} \)</td>
    <td>17.18\pm2.01</td>
    <td>12.82\pm0.15</td>
    <td>21.79\pm5.20</td>
  </tr>
  <tr>
    <td>+ MG-OCG trained on \( D^{train} \)</td>
    <td><b>21.18\pm1.07</b></td>
    <td>18.70\pm0.73</td>
    <td>25.43\pm5.28</td>
  </tr>
  <tr>
    <td>+ LongGuide trained on \( D^{train} \)</td>
    <td><b>21.18\pm1.07</b></td>
    <td><b>19.86\pm2.93</b></td>
    <td>25.05\pm0.76</td>
  </tr>
  <tr>
    <td>- + OCG trained on Demos</td>
    <td>16.88\pm1.44</td>
    <td>19.40\pm1.39</td>
    <td><b>28.28\pm7.69</b></td>
  </tr>
  <tr>
    <td>+ MG trained on Demos</td>
    <td>15.59\pm0.59</td>
    <td>12.07\pm2.68</td>
    <td>23.99\pm4.66</td>
  </tr>
  <tr>
    <td>+ MG-OCG trained on Demos</td>
    <td>19.89\pm0.39</td>
    <td>17.78\pm3.23</td>
    <td>27.41\pm0.87</td>
  </tr>
  <tr>
    <td>+ LongGuide trained on Demos</td>
    <td>19.89\pm0.39</td>
    <td>17.78\pm18.43</td>
    <td>23.99\pm4.66</td>
  </tr>
</table>

Table 15: LongGuide learns the guidelines from only demonstrations with Mistral, evaluated by ROUGE-L.

![Bar chart showing fine-grained human evaluation results on evaluated MG metrics](page_1012_1342_1022_312.png)

Figure 11: Fine-grained human evaluation results on evaluated MG metrics.
<table>
  <tr>
    <th></th>
    <th>MG</th>
    <th>Context (MG)</th>
    <th>OCG</th>
    <th>Context (OCG)</th>
    <th>MG-OCG</th>
    <th>Context (MG-OCG)</th>
  </tr>
  <tr>
    <td>Per token</td>
    <td>0.0019</td>
    <td>0.0064</td>
    <td>0.0133</td>
    <td>0.0077</td>
    <td>0.0017</td>
    <td>0.0064</td>
  </tr>
  <tr>
    <td>All</td>
    <td>37.81%</td>
    <td>62.19%</td>
    <td>22.56%</td>
    <td>77.44%</td>
    <td>37.87%</td>
    <td>62.13%</td>
  </tr>
</table>

Table 16: Attention score over guideline and context tokens of Mistral.

E.10 Human Evaluation Fine-grained Results (§5.1)

Figure 11 presents our fine-grained human evaluation results. Overall, LongGuide shows the best in terms of “Accuracy” and “Clarity”, with a significant number of winning ratings. This suggests that the generated text is factually correct and easy to understand. Meanwhile, LongGuide shows more mixed results in terms of “Clarity” and “Coherence”. While there is still a high winning rating, the proportion of draw and loss ratings is also relatively high, possibly because improving “Brevity” can somehow reduce the “Clarity”.

E.11 Attention Analysis for Guideline Tokens (§E.2)

Table 16 shows our simple attention analysis.

<table>
  <tr>
    <th>Task</th>
    <th>Model</th>
    <th>Selected Metrics</th>
  </tr>
  <tr>
    <td>SAMSum</td>
    <td>Mistral<br>ChatGPT</td>
    <td>[‘Accuracy’, ‘Brevity’, ‘Clarity’, ‘Relevance’, ‘Understandability’]<br>[‘Accuracy’, ‘Brevity’, ‘Clarity’, ‘Relevance’, ‘Understandability’]</td>
  </tr>
  <tr>
    <td>CNN</td>
    <td>Mistral<br>ChatGPT</td>
    <td>[‘Accuracy’, ‘Brevity’, ‘Clarity’, ‘Coherence’, ‘Completeness’, ‘Engagement’, ‘Readability’, ‘Relevance’, ‘Truthfulness’, ‘Understandability’]<br>[‘Accuracy’, ‘Brevity’, ‘Clarity’, ‘Coherence’, ‘Completeness’, ‘Conciseness’, ‘Engagement’, ‘Neutrality’, ‘Readability’, ‘Relevance’, ‘Specificity’]</td>
  </tr>
  <tr>
    <td>XLSum</td>
    <td>Mistral<br>ChatGPT</td>
    <td>[‘Accuracy’, ‘Brevity’, ‘Clarity’, ‘Coherence’, ‘Completeness’, ‘Consistency’, ‘Correctness’, ‘Diversity’, ‘Engagement’, ‘Factuality’, ‘Fluency’, ‘Indicative’, ‘Informativeness’, ‘Neutrality’, ‘Non-repetitiveness’, ‘Relevance’, ‘Resolution’, ‘Respect of Chronology’, ‘Semantic Coverage’, ‘Specificity’, ‘Understandability’]<br>[‘Accuracy’, ‘Brevity’, ‘Clarity’, ‘Coherence’, ‘Completeness’, ‘Consistency’, ‘Correctness’, ‘Diversity’, ‘Engagement’, ‘Factuality’, ‘Fluency’, ‘Indicative’, ‘Informativeness’, ‘Neutrality’, ‘Non-repetitiveness’, ‘Rationalness’, ‘Relevance’, ‘Resolution’, ‘Respect of Chronology’, ‘Semantic Coverage’, ‘Specificity’, ‘Understandability’]</td>
  </tr>
  <tr>
    <td>SWiPE</td>
    <td>Mistral<br>ChatGPT</td>
    <td>[‘Accuracy’, ‘Brevity’, ‘Clarity’, ‘Relevance’, ‘Understandability’]<br>[‘Accuracy’, ‘Brevity’, ‘Clarity’, ‘Coherence’, ‘Conciseness’, ‘Consistency’, ‘Correctness’, ‘Readability’, ‘Understandability’]</td>
  </tr>
  <tr>
    <td>IWSLT17 en-ja</td>
    <td>Mistral<br>ChatGPT</td>
    <td>[‘Accuracy’, ‘Clarity’, ‘Coherence’, ‘Consistency’, ‘Correctness’, ‘Factuality’, ‘Fluency’, ‘Relevance’, ‘Understandability’]<br>[‘Accuracy’, ‘Clarity’, ‘Coherence’, ‘Consistency’, ‘Correctness’, ‘Factuality’, ‘Fluency’, ‘Relevance’, ‘Understandability’]</td>
  </tr>
  <tr>
    <td>Synthetic Persona</td>
    <td>Mistral<br>ChatGPT</td>
    <td>[‘Accuracy’, ‘Brevity’, ‘Clarity’, ‘Coherence’, ‘Completeness’, ‘Consistency’, ‘Correctness’, ‘Diversity’, ‘Engagement’, ‘Factuality’, ‘Fluency’, ‘Indicative’, ‘Informativeness’, ‘Neutrality’, ‘Non-repetitiveness’, ‘Relevance’, ‘Resolution’, ‘Respect of Chronology’, ‘Semantic Coverage’, ‘Specificity’, ‘Understandability’]<br>[‘Accuracy’, ‘Clarity’, ‘Coherence’, ‘Consistency’, ‘Correctness’, ‘Diversity’, ‘Engagement’, ‘Fluency’, ‘Indicative’, ‘Informativeness’, ‘Neutrality’, ‘Non-repetitiveness’, ‘Relevance’, ‘Resolution’, ‘Respect of Chronology’, ‘Specificity’, ‘Understandability’]</td>
  </tr>
  <tr>
    <td>CommGen-Chall.</td>
    <td>Mistral<br>ChatGPT</td>
    <td>[‘Coherence’, ‘Conciseness’, ‘Fluency’, ‘Relevance’, ‘Understandability’]<br>[‘Clarity’, ‘Coherence’, ‘Completeness’, ‘Conciseness’, ‘Consistency’, ‘Creativity’, ‘Engagement’, ‘Fluency’, ‘Naturalness’, ‘Relevance’]</td>
  </tr>
</table>

Table 17: Selected metrics by tasks by Mistral and ChatGPT.

E.12 Which Metrics Were Selected the Most for MG? (§E.5)

To better understand how models select and evaluate metrics, we analyze the specific metrics chosen for each task (Table 17), their selection frequencies (Figure 12), and their average scores (Figure 13).
Figure 12: Frequency of metrics selected as the metric guideline.

Figure 13: Average scores of metrics as the metric guideline.

<table>
  <tr>
    <th>Methods</th>
    <th>CNN (3.0.0)</th>
    <th>IWSLT17 en-ja</th>
    <th>CommGen-Chall.</th>
  </tr>
  <tr>
    <td>Zero-shot (ZS)</td>
    <td>19.23±0.34</td>
    <td>13.12±1.39</td>
    <td>10.12±0.02</td>
  </tr>
  <tr>
    <td>+ LongGuide</td>
    <td><b>22.46±0.64</b></td>
    <td><b>16.53±0.59</b></td>
    <td><b>25.20±1.89</b></td>
  </tr>
  <tr>
    <td>+ LongGuide w/o Token Constraint</td>
    <td>21.54±0.52</td>
    <td>14.09±1.07</td>
    <td>21.49±2.15</td>
  </tr>
  <tr>
    <td>+ LongGuide w/o Sentence Constraint</td>
    <td>20.92±0.23</td>
    <td>10.02±1.17</td>
    <td>13.32±0.73</td>
  </tr>
  <tr>
    <td>Few-shot (FS)</td>
    <td>17.56±0.63</td>
    <td>12.69±1.82</td>
    <td>3.98±0.17</td>
  </tr>
  <tr>
    <td>+ LongGuide</td>
    <td><b>21.18±1.07</b></td>
    <td><b>19.86±2.93</b></td>
    <td><b>25.05±0.76</b></td>
  </tr>
  <tr>
    <td>+ LongGuide w/o Token Constraint</td>
    <td>20.30±1.46</td>
    <td>19.75±1.47</td>
    <td>20.30±1.46</td>
  </tr>
  <tr>
    <td>+ LongGuide w/o Sentence Constraint</td>
    <td>15.89±2.26</td>
    <td>12.57±2.99</td>
    <td>12.20±3.91</td>
  </tr>
</table>

Table 18: Mistral results when omitting OCG’s Token or Sentence Information, showing the importance of OCG’s Token and Sentence information, evaluated by ROUGE-L.
E.13 Extra Ablation Studies: without OCG’s Token or Sentence Constraint (§E.5)

Since OCG’s token information and sentence information are the two types of information emphasized in OCG, we further investigate the importance of each type of information. The empirical experiments are conducted with Mistral on CNN, IWSLT-2017 en-ja, and CommonGen-Challenge. We present the results in Table 18. We observe that skipping OCG’s token information or sentence information would hurt the performance. Specifically, the results drop more significantly when sentence information is omitted, and even fall below the Zero-shot score in CNN Few-shot with LongGuide and IWSLT17 en-ja Few-shot with LongGuide. The performance drops significantly in the CommonGen-Challenge Few-shot case, with a fall of 55.20%. Due to the volatility of the token count in a sentence, it is hard to estimate the other information with only one type of information given. Therefore, both types of information should be provided to better capture the text distribution.

F Implementation Details

Task benchmark preprocessing. We chose the newest versions of the above datasets. For each dataset except Synthetic-Persona-Chat, we sample 200 samples from the test set for our evaluation, following Bai et al. (2024), and 50 random samples from the train set for \( D^{train} \). For Synthetic-Persona-Chat, we randomly sample 25 dialogues from its test set for our evaluation (678 utterances in total) and 3 dialogues from its train set where 50 random utterances are selected for \( D^{train} \).

Prompting baselines’ hyperparameters. We present the implementation and hyperparameters’ details for our proposed LongGuide as well as prompting baselines below.

• LongGuide. We set the batch size is 5 and number of iterations is also 5 for LongGuide’s step 1. For steps 2, 3, and 4, no hyperparameter involves. For the evaluations by Self-consistency (Wang et al., 2022a), we sample 3 results.

• APO (Pryzant et al., 2023). We set the number of optimization iterations is 5. We use 1 sample with the lowest ROUGE-L score as the error sample for generating gradients, following (Do et al., 2024). At each iteration, 5 textual gradients are generated, and 5 new prompts are sampled from textual gradients. Finally, 1 paraphrase of the input prompt is sampled at each optimization iteration.

• adv-ICL (Do et al., 2024). We use 3 iterations with a batch size of 5 as suggested by (Do et al., 2024). At each iteration, the number of new prompts sampled is 5.

Models’ hyperparameters. The models’ hyperparameters are presented below.

• ChatGPT. We use gpt-3.5-turbo-1106 for our experiments. We use a window size of 1500 and Nucleus Sampling (Holtzman et al., 2019) as our decoding strategy with a \( p \) value of 1. We use the system role as “You are a helpful assistant!”.

• Mistral-7B-it-v0.2. We use a window size of 1500, and Sampling decoding strategy (Holtzman et al., 2019) (\( do_sampling = True \)). We load the model from Huggingface Transformers library (Wolf et al., 2020) with the model id is “mistralai/Mistral-7B-Instruct-v0.2”. We do not set any explicit system role.

G Prompts and Prompting Analysis

G.1 GPT-4o-Judge’s Prompt

Our GPT-4o-Judge prompt evaluating the generated response and the reference is heavily motivated by Zheng et al. (2023).
Please act as an impartial judge and evaluate how well an assistant’s answer aligns with the reference answer and the quality of the assistant’s answer. You will be given a user prompt, a reference answer and an assistant’s answer. Your evaluation must consider the following criteria:
- Format consistency: ensuring the generated response matches the length and structure of the reference.
- Content completeness: evaluating whether all key points present in the reference are included in the assistant’s answer.
- Factuality: checking for factual correctness of the assistant’s answer.
- Style adherence: ensuring that the tone, style, and level of detail of the of the assistant’s answer match the reference.
- Assistant’s answer quality: assessing how well the response satisfies the user’s requirements.
Begin your evaluation by providing a short explanation for each. Be as objective as possible. After providing your explanation, please rate the response on all the criterion on a scale of 1 to 10 by strictly following this format:
[The Start of Explanation]
...
[The End of Explanation]
[The Start of Ratings]
{
"Format": 1-10,
"Content": 1-10,
"Factuality": 1-10,
"Style": 1-10,
"Quality": 1-10,
}
[The End of Ratings]
[User Prompt]
user_prompt
[The Start of Reference Answer]
answer_ref
[The End of Reference Answer]
[The Start of Assistant’s Answer]
answer_a
[The End of Assistant’s Answer]
G.2 ChatGPT Property Scorer Prompt

You are an expert in evaluating the quality of a text generation task. You possess a nuanced understanding of various critical aspects. Brevity is paramount for you, ensuring concise expression without sacrificing essential information. Clarity is essential for comprehension, ensuring that your text is easily understood by the intended audience. Relevance ensures that the generated content aligns closely with the given context or prompt. Neutrality is crucial, maintaining an impartial tone devoid of bias. Coherence ties together ideas seamlessly, fostering a logical flow within your text. Completeness guarantees that all relevant points are addressed adequately. Specificity enhances precision, providing detailed and accurate information. Respect of chronology ensures temporal coherence, maintaining the chronological order of events. Accuracy demands factual correctness, avoiding errors or misinformation. Non-repetitiveness prevents redundancy, ensuring freshness in your expression. Indicative language aids in signaling key points or conclusions. Lastly, resolution ensures that your text concludes satisfactorily, resolving any questions or issues raised throughout.

Input: {dialogue}
Output: {generated_summary}
Your task is to evaluate the following criteria in a scale of 1-5, with 1 is worst and 5 is best.
{
"Semantic Coverage": 1-5,
"Factuality": 1-5,
"Consistency": 1-5,
"Informativeness": 1-5,
"Coherence": 1-5,
"Relevance": 1-5
}
The definitions of the criteria are:
Semantic Coverage (COV): The extent to which a dialogue summary captures the main ideas and topics discussed in the conversation.
Factuality (FAC): The accuracy and truthfulness of the information presented in the dialogue summary, reflecting fidelity to the original conversation.
Consistency (CON): The degree to which the summary maintains logical and contextual coherence throughout, avoiding contradictory or conflicting information.
Informativeness (INF): The richness and depth of information conveyed in the dialogue summary, including key details and relevant context.
Coherence (COH): The overall clarity and organization of the summary, ensuring smooth transitions between ideas and coherence in the narrative flow.
Relevance (REL): The pertinence of the information included in the dialogue summary to the intended purpose or topic, ensuring alignment with the user’s interests or needs.
Your output must be in Python dictionary format.

G.3 LongGuide’s Prompts

Prompting templates for LongGuide. Let \( Q, C, I, D_f \) be the input query, context, instruction, and demonstration token sequence respectively (\S1, \S2), and \( G^{best} \) is the learned guideline(s), the prompt for \( \mathcal{M} \) is formatted: “\{I\}\n\{D_f\}\n\{C\}\n\{Q\}\n\{G^{best}\}”.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Method</th>
    <th colspan="4">Summarization</th>
    <th colspan="2">Simplification</th>
    <th colspan="2">Translation</th>
    <th colspan="2">Dialogue Generation</th>
    <th colspan="2">Table2Text</th>
  </tr>
  <tr>
    <th>SAMSum</th>
    <th>CNN (3.8.0)</th>
    <th>XL-Sum</th>
    <th>SWPE</th>
    <th>IWSLT17 en-ja</th>
    <th>Synthetic Persona</th>
    <th>CommGen-Chall.</th>
  </tr>
  <tr>
    <td rowspan="2">Metric</td>
    <td>#shots (random)</td>
    <td>3</td>
    <td>3</td>
    <td>5</td>
    <td>3</td>
    <td>5</td>
    <td>5</td>
    <td>5</td>
    <td>5</td>
  </tr>
  <tr>
    <td>#tokens consumed</td>
    <td>642</td>
    <td>1110</td>
    <td>811</td>
    <td>1020</td>
    <td>915</td>
    <td>855</td>
    <td>939</td>
  </tr>
  <tr>
    <td rowspan="2">Metric</td>
    <td>USS consumed</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>ChatGPT</td>
    <td>#tokens consumed</td>
    <td>1866</td>
    <td>7683</td>
    <td>4863</td>
    <td>2380</td>
    <td>1370</td>
    <td>1344</td>
    <td>1272</td>
  </tr>
  <tr>
    <td></td>
    <td>USS consumed</td>
    <td>insignificant</td>
    <td>insignificant</td>
    <td>insignificant</td>
    <td>insignificant</td>
    <td>insignificant</td>
    <td>insignificant</td>
    <td>insignificant</td>
  </tr>
</table>

Table 19: Total number of tokens consumed and USS consumed for models to learn the metric guideline (MG) and output constraint guideline (OCG).

Prompting costs. Table 19 presents the total number of tokens consumed for models to learn the metric guidelines and output constraint guideline (OCG) for both models with the hyperparameters of LongGuide specified in \S F. We observe that the number of tokens needed to learn the guidelines is insignificant, demonstrating that LongGuide is a cost-effective solution and potentially beneficial for a wide range of applications.

Table 20 presents the prompting cost comparision between LongGuide and other PO algorithms. We compare the number of new prompts sampled by each algorithm for validation set verification, as these
prompts are the primary cost bottleneck in PO algorithms. We observe that LongGuide is approximately at least \(3.75\) times cheaper than adv-ICL in both settings and \(18.75\) times cheaper than APO. For SAMSum, the validation of one prompt using 50 samples involves approximately 22K tokens, which incurs a cost of 0.02 USD as of November 19, 2024.

Prompt for step 1, metric selection. Below is the prompt we use for step 1 selecting metrics for a given task.

Select top-5 metrics that are the most important from the list below to evaluate a special way of {TASK_NAME}. {str(PRE_DEFINED_ASSESSMEN_METRICS)}.
Here are some demonstrations of the task {TASK_NAME}: {DEMONSTRATION_STRING}.
Output your list of metrics in Python list format without any explanation: [...].

Prompt for step 2, metric score collection. Below is the prompt we use for step 2 for evaluating selected metrics on the task.

You are given an input and an output of a {TASK_NAME} task.
Input: {input}
Output: {output}
Your task is to evaluate the following criteria on a scale of 1-5, with 1 being worst and 5 being best.
{EVALUATION_FORMAT}
The definitions of the criteria are: {METRICS_DEFINITIONS}
Your output must be in Python dictionary format without explanation.

Prompt for step 2, collecting metrics’ definitions. Below is the prompt we use for step 2 collecting METRICS_DEFINITIONS for step 2.

Define the list of following metrics in details as the quality of the output expected for the {TASK_NAME} task.
{metrics}
Give me the list in bullet points.

Prompt for step 3, generating metric guideline (MG). Below is the prompt we use for step 3, generating the metric guideline (MG).

Now you are given the following metrics: {metrics_string} for the {TASK_NAME} task.
Based on these scores on a scale of 5 for the quality of the output: {str(metrics_collected_scores)}, define the expected quality of the output for each metric in natural language. Give me the list in bullet points.

H Examples
<table>
  <tr>
    <th>Method</th>
    <th>#Prompts Sampled</th>
    <th>Cost</th>
  </tr>
  <tr>
    <td>ZS</td>
    <td>adv-ICL (3 iterations) × (1 instruction) × (5 variants)<br>APO (5 iterations) × (15 prompts sampled) × (1 instruction)<br>LongGuide 4 prompts (MG, OCG, MG-OCG, No guideline)</td>
    <td>15 × prompt validation cost<br>75 × prompt validation cost<br><b>4 × prompt validation cost</b></td>
  </tr>
  <tr>
    <td>FS</td>
    <td>adv-ICL (3 iterations) × (3 demonstrations + 1 instruction) × (5 variants)<br>APO (5 iterations) × (15 prompts sampled) × (3 demonstrations + 1 instruction)<br>LongGuide 4 prompts (MG, OCG, MG-OCG, No guideline)</td>
    <td>60 × prompt validation cost<br>300 × prompt validation cost<br><b>4 × prompt validation cost</b></td>
  </tr>
</table>

Table 20: Prompting cost comparison between PO methods and LongGuide based on # new prompts sampled to test over the validation set.

Zero-shot + OCG (ROUGE-L = 29.85)

Summarize the summary from the following dialogue.

Morgan: Hey gorgeous, how's your day?
Suzanne: Nothing special, it's just one of many boring days at work. But... better now though!
Morgan: Are you working at all? 😊
Suzanne: I'm trying 😊 But you aren't helping me, at all
Suzanne: I'm just taking a well-deserved break 😊
Morgan: I miss you Suzie
Suzanne: I miss you too Morgan
Morgan: Do you feel like going to a concert next week? Maroon 5 is playing at the Hulu Theater at Madison Square Garden.
Morgan: As it happens, I've got two tickets 😊
Morgan: Do you want to go?
Suzanne: Really? OMG! That's wonderful!
Suzanne: Thank you sweetheart!
Morgan: Oh, nothing. I just want you to be happy 😊

The summary must have from 1 to 4 sentences and from 5 to 51 words with an average of 22 words and 1 sentences.

ChatGPT: Morgan and Suzanne are talking about their day at work. Morgan offers Suzanne tickets to a Maroon 5 concert, and Suzanne expresses excitement and thanks Morgan. They plan to attend the concert together.

Few-shot (3 shots) + MG-OCG (ROUGE-L = 40.74)

Summarize the summary from the following dialogue.

Your generated summary must strictly fulfill the following task metrics
- Accuracy: The summary is completely and precisely capturing the essence and key points of the original dialogue with no omissions or errors.
- Clarity: The summary is clear, easy to understand, and free from ambiguity or confusion.
- Coherence: The summary maintains logical and consistent flow of ideas and concepts, making it easy to follow.
- Completeness: The summary includes all essential information from the original dialogue without missing any critical details.
- Understandability: The summary is straightforward and accessible, making it comprehendible to a broad audience.
- Brevity: The summary is succinct and to the point, conveying the most important ideas using the least possible number of words.
- Relevance: The summary focuses on the main points and ideas of the original dialogue that are important and significant to the task or context.

The summary must have from 1 to 4 sentences and from 5 to 51 words with an average of 22 words and 1 sentences.

Input: [Demonstration 1 dialogue]
Output: [Demonstration 1 summary]

Input: [Demonstration 2 dialogue]
Output: [Demonstration 2 summary]

Input: [Demonstration 3 dialogue]
Output: [Demonstration 3 summary]

Input:
Morgan: Hey gorgeous, how's your day?
Suzanne: Nothing special, it's just one of many boring days at work. But... better now though!
Morgan: Are you working at all? 😊
Suzanne: I'm trying 😊 But you aren't helping me, at all
Suzanne: I'm just taking a well-deserved break 😊
Morgan: I miss you Suzie
Suzanne: I miss you too Morgan
Morgan: Do you feel like going to a concert next week? Maroon 5 is playing at the Hulu Theater at Madison Square Garden.
Morgan: As it happens, I've got two tickets 😊
Morgan: Do you want to go?
Suzanne: Really? OMG! That's wonderful!
Suzanne: Thank you sweetheart!
Morgan: Oh, nothing. I just want you to be happy 😊

Output:

ChatGPT: Morgan invites Suzanne to a Maroon 5 concert at Madison Square Garden and she gladly accepts, expressing happiness and gratitude

Figure 14: Full example of ChatGPT results on a SAMSum example (Gliwa et al., 2019) w/ LongGuide guidelines (Zero-shot + LongGuide and Few-shot + LongGuide) from Figure 1
Ground-truth Answer (Summary)
Rob and Eve will meet on Sunday morning to go to the shops. Eve has something to do at about 3

Zero-shot (ROUGE-L = 30.77)

Summarize the summary from the following dialogue.

Rob: Are we meeting up 2morrow?
Eve: How about Sunday
Eve: stores are open so we can go then
Rob: Ok I just don't know what time they close
Eve: we have ot go in the moring
Eve: I have some stuff to do around 3
Rob: ok

ChatGPT: Rob and Eve discuss meeting up, with Eve suggesting they go on Sunday since stores are open. Rob agrees but is concerned about store closing times. Eve suggests going in the morning as she has plans in the afternoon. Rob agrees to the plan.

Zero-shot + MG (ROUGE-L = 36.07)

Summarize the summary from the following dialogue.

Rob: Are we meeting up 2morrow?
Eve: How about Sunday
Eve: stores are open so we can go then
Rob: Ok I just don't know what time they close
Eve: we have ot go in the moring
Eve: I have some stuff to do around 3
Rob: ok

Your generated summary must strictly fulfill the following task metrics
- Accuracy: The summary is completely and precisely capturing the essence and key points of the original dialogue with no omissions or errors.
- Clarity: The summary is clear, easy to understand, and free from ambiguity or confusion.
- Coherence: The summary maintains a logical and consistent flow of ideas and concepts, making it easy to follow.
- Completeness: The summary includes all essential information from the original dialogue without missing any critical details.
- Understandability: The summary is straightforward and accessible, making it comprehensible to a broad audience.
- Brevity: The summary is succinct and to the point, conveying the most important ideas using the least possible number of words.
- Relevance: The summary focuses on the main points and ideas of the original dialogue that are important and significant to the task or context.

ChatGPT: Rob and Eve discuss meeting up, considering Sunday due to stores being open. They plan to go in the morning as Eve has plans later. Rob expresses uncertainty about store closing times, to which Eve suggests going in the morning.

Zero-shot + OCG (ROUGE-L = 36.73)

Summarize the summary from the following dialogue.

Rob: Are we meeting up 2morrow?
Eve: How about Sunday
Eve: stores are open so we can go then
Rob: Ok I just don't know what time they close
Eve: we have ot go in the moring
Eve: I have some stuff to do around 3
Rob: ok

The summary must have from 1 to 4 sentences and from 5 to 51 words with an average of 22 words and 1 sentences.

ChatGPT: Rob and Eve plan to meet on Sunday when stores are open, but Rob is unsure about the closing time, so they decide to go in the morning.

Zero-shot + MG-OCG (ROUGE-L = 44.00)

Summarize the summary from the following dialogue.

Rob: Are we meeting up 2morrow?
Eve: How about Sunday
Eve: stores are open so we can go then
Rob: Ok I just don't know what time they close
Eve: we have ot go in the moring
Eve: I have some stuff to do around 3
Rob: ok

Your generated summary must strictly fulfill the following task metrics
- Accuracy: The summary is completely and precisely capturing the essence and key points of the original dialogue with no omissions or errors.
- Clarity: The summary is clear, easy to understand, and free from ambiguity or confusion.
- Coherence: The summary maintains a logical and consistent flow of ideas and concepts, making it easy to follow.
- Completeness: The summary includes all essential information from the original dialogue without missing any critical details.
- Understandability: The summary is straightforward and accessible, making it comprehensible to a broad audience.
- Brevity: The summary is succinct and to the point, conveying the most important ideas using the least possible number of words.
- Relevance: The summary focuses on the main points and ideas of the original dialogue that are important and significant to the task or context.

The summary must have from 1 to 4 sentences and from 5 to 51 words with an average of 22 words and 1 sentences.

ChatGPT: Rob and Eve plan to meet on Sunday to take advantage of store hours; they aim to go in the morning, allowing Eve time for personal activities around 3.

Figure 15: A sample from SAMSum dataset where MG and OCG supplement each other and are not interchangeable to increase the performance in final answer.
Simplify the following text

Yehuda Nir (March 31, 1930 – July 19, 2014) was a Polish-born American Holocaust survivor, psychiatrist and author of The Lost Childhood. Nir posed as a Roman Catholic and learned Latin to escape Nazi persecution in Poland during World War II. Nir’s ordeal led him to a career as a psychiatrist, specializing in the treatment of post-traumatic stress disorder and severely ill children. He immigrated to the United States in 1959 to complete medical residencies in New York City and Philadelphia. He served as the chief of child psychiatry at Memorial Sloan-Kettering Cancer Center from 1979 until 1986.

Nir was born Juliusz Grunfeld in Lvov, Poland, (present-day Ukraine) on March 31, 1930. He later changed his name to “Nir” after World War II since “Grunfeld” has German origins. Nir means plowed fields in Hebrew.

Nir released a memoir of his experience during the Holocaust, “The Lost Childhood” in 1989. A second edition was reprinted by Scholastic Press in 2002. The Lost Childhood is now used as part of the high school curriculum throughout the United States. He also published four self-help books focusing on relationships, including “Not Quite Paradise: Making Marriage Work” and “Loving Men for All the Right Reasons.”

Yehuda Nir died at his home in Manhattan, New York City, on July 19, 2014, at the age of 84. His funeral was held at Riverside Memorial Chapel on the Upper West Side with burial on the Mount of Olives in Jerusalem.

ChatGPT: Yehuda Nir, born Juliusz Grunfeld in Lvov, Poland (now Ukraine) in 1930, survived the Holocaust by posing as a Roman Catholic and learning Latin. He later became a psychiatrist, focusing on PTSD and severely ill children in the US. Nir published “The Lost Childhood” about his experiences and self-help books on relationships. He died in 2014, with a funeral in New York and burial in Jerusalem.

Born in Lvov, Poland, in 1930 as Juliusz Grunfeld, Yehuda Nir posed as a Catholic during the Holocaust and used his knowledge of Latin to hide. After immigration to the US in 1959, he specialized in child psychiatry. Nir published his memoir, “The Lost Childhood,” in 1989 and it is now used in US high schools. Additionally, he wrote four self-help books, including “Not Quite Paradise” and “Loving Men for All the Right Reasons.” Yehuda Nir, aged 84, passed away in Manhattan, New York, in 2014. His funeral was held at Riverside Memorial Chapel and he was buried on the Mount of Olives in Jerusalem. (118 tokens)

Zero-shot + OCG (ROUGE-L = 15.79)

Simplify the following text

Yehuda Nir (March 31, 1930 – July 19, 2014) was a Polish-born American Holocaust survivor, psychiatrist and author of The Lost Childhood. Nir posed as a Roman Catholic and learned Latin to escape Nazi persecution in Poland during World War II. Nir’s ordeal led him to a career as a psychiatrist, specializing in the treatment of post-traumatic stress disorder and severely ill children. He immigrated to the United States in 1959 to complete medical residencies in New York City and Philadelphia. He served as the chief of child psychiatry at Memorial Sloan-Kettering Cancer Center from 1979 until 1986.

Nir was born Juliusz Grunfeld in Lvov, Poland, (present-day Ukraine) on March 31, 1930. He later changed his name to “Nir” after World War II since “Grunfeld” has German origins. Nir means plowed fields in Hebrew.

Nir released a memoir of his experience during the Holocaust, “The Lost Childhood” in 1989. A second edition was reprinted by Scholastic Press in 2002. The Lost Childhood is now used as part of the high school curriculum throughout the United States. He also published four self-help books focusing on relationships, including “Not Quite Paradise: Making Marriage Work” and “Loving Men for All the Right Reasons.”

Yehuda Nir died at his home in Manhattan, New York City, on July 19, 2014, at the age of 84. His funeral was held at Riverside Memorial Chapel on the Upper West Side with burial on the Mount of Olives in Jerusalem.

The simplified text must have from 1 to 17 sentences and from 16 to 426 words with an average of 80 words and 4 sentences.

ChatGPT: Yehuda Nir, born Juliusz Grunfeld in Poland in 1930, survived the Holocaust by posing as a Roman Catholic and learned Latin. He later became a psychiatrist, focusing on PTSD and severely ill children in the US. Nir published “The Lost Childhood” about his experiences and self-help books on relationships. He died in 2014, with a funeral in New York and burial in Jerusalem.

Born in Lvov, Poland, in 1930 as Juliusz Grunfeld, Yehuda Nir posed as a Catholic during the Holocaust and used his knowledge of Latin to hide. After immigration to the US in 1959, he specialized in child psychiatry. Nir published his memoir, “The Lost Childhood,” in 1989 and it is now used in US high schools. Additionally, he wrote four self-help books, including “Not Quite Paradise” and “Loving Men for All the Right Reasons.” Yehuda Nir, aged 84, passed away in Manhattan, New York, in 2014. His funeral was held at Riverside Memorial Chapel and he was buried on the Mount of Olives in Jerusalem. (117 tokens)

Zero-shot + MG (ROUGE-L = 21.82)

Simplify the following text

Yehuda Nir (March 31, 1930 – July 19, 2014) was a Polish-born American Holocaust survivor, psychiatrist and author of The Lost Childhood. Nir posed as a Roman Catholic and learned Latin to escape Nazi persecution in Poland during World War II. Nir’s ordeal led him to a career as a psychiatrist, specializing in the treatment of post-traumatic stress disorder and severely ill children. He immigrated to the United States in 1959 to complete medical residencies in New York City and Philadelphia. He served as the chief of child psychiatry at Memorial Sloan-Kettering Cancer Center from 1979 until 1986.

Nir was born Juliusz Grunfeld in Lvov, Poland, (present-day Ukraine) on March 31, 1930. He later changed his name to “Nir” after World War II since “Grunfeld” has German origins. Nir means plowed fields in Hebrew.

Nir released a memoir of his experience during the Holocaust, “The Lost Childhood” in 1989. A second edition was reprinted by Scholastic Press in 2002. The Lost Childhood is now used as part of the high school curriculum throughout the United States. He also published four self-help books focusing on relationships, including “Not Quite Paradise: Making Marriage Work” and “Loving Men for All the Right Reasons.”

Yehuda Nir died at his home in Manhattan, New York City, on July 19, 2014, at the age of 84. His funeral was held at Riverside Memorial Chapel on the Upper West Side with burial on the Mount of Olives in Jerusalem.

The simplified text must have from 1 to 17 sentences and from 16 to 426 words with an average of 80 words and 4 sentences.

ChatGPT: Dr. Yehuda Nir, born Juliusz Grunfeld in 1930 in Lvov, Poland, survived the Holocaust by posing as a Catholic. He moved to the U.S. in 1959, becoming a psychiatrist specializing in PTSD and child psychiatry. His memoir, The Lost Childhood, published in 1989, is used in high schools. Nir died in Manhattan in 2014, with his burial in Jerusalem. (99 tokens)

Figure 16: An example of SWiPE (Laban et al., 2023) where the record contains fewer tokens than the expected average. This reduces the effectiveness of OCG and MG individually, but their combination could enhance performance.
<table>
  <tr>
    <th>Input (Concepts)</th>
    <th>Ground truth (Commonsense description)</th>
    <th>Metric guideline scores</th>
    <th>Answer</th>
  </tr>
  <tr>
    <td>['building', 'hang', 'wash', 'washer', 'window']</td>
    <td>The lady takes the wash out of the washer and hangs them out the buildings window to dry.</td>
    <td>Consiseness: 4<br>Informativeness: 5<br>...</td>
    <td>When it comes to maintaining a clean building, having a hang washer can be incredibly useful. This tool makes it easier to wash and clean windows, particularly in high-rise structures where traditional methods may not be feasible. The hang washer allows for efficient cleaning and maintenance, ensuring that the windows are kept pristine and the building maintains a pristine appearance. By using a hang washer, building maintenance can be both effective and safe, providing a reliable solution for keeping windows clean and ensuring the overall upkeep of the structure.</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td>❌ Conciseness   ✔️ Informativeness</td>
  </tr>
  <tr>
    <td>Input (Concepts)</td>
    <td>Ground truth (Commonsense description)</td>
    <td>Metric guideline scores</td>
    <td>Answer</td>
  </tr>
  <tr>
    <td>['ball', 'hit', 'take', 'turn']</td>
    <td>So there I am at the party, standing on the second floor looking at John, so I hit the ball against his face, then I turn around and put it in my hand and take a leap of faith that John won't suspect of me.</td>
    <td>Consiseness: 5<br>Informativeness: 3<br>...</td>
    <td>The ball was hit, and then it was taken by the player's turn.</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td>✔️ Conciseness   ❌ Informativeness</td>
  </tr>
</table>

Figure 17: A CommonGen-Challenge example (Lin et al., 2020), where output with high Conciseness score could have low Informativeness score and vice versa

Input (Dialogue)
George: Hi Donna. I've been trying to catch you.
Donna: What about?
George: A rather delicate matter.
Donna: Did you catch AIDS?
George: Very funny!
Donna: It is, isn't it?
George: I don't think so.
Donna: Too bad. So what do you want?
George: Could we meet and discuss it somewhere?
Donna: Where were?
George: Like in a coffee shop or somewhere.
Donna: You sure? Coffee shop is better to discuss delicate matters.
George: Come to think of it, you are right. It's not.
Donna: See? So what's up?
George: Couldn't you come to my place tonight?

Ground truth answer (Summary)
Donna will pay George a visit tonight to discuss a personal matter.

Zero-shot answer (ROUGE-L = 16.22)
George attempts to make arrangements to meet Donna to discuss a personal matter, but she is skeptical and makes a joke about the topic being AIDS. George finds her humor inappropriate and denies it. Donna presses for information on the discussion topic, and George suggests they meet at his place that night since a coffee shop may not be an appropriate setting.

LongGuide (ROUGE-L = 21.62)
(MG metrics' scores: Accuracy: 5, Brevity: 5, Clarity: 4, Correctness: 5, Relevance: 5, Understandability: 5, Informativeness: 4)
George proposed discussing a private matter with Donna, suggesting a coffee shop but later acknowledging its unsuitability; he then invited her to his place instead.

LongGuide without Step 2 (ROUGE-L = 17.24)
(MG metrics' scores w/o step 2: Accuracy: 5, Brevity: 5, Clarity: 5, Correctness: 5, Relevance: 5, Understandability: 5, Informativeness: 5)
George asked Donna to discuss a delicate matter with him, initially suggesting a coffee shop. However, Donna pointed out that a coffee shop might not be the best place for such a conversation, leading George to reconsider and propose coming to his place that night instead.

Figure 18: A SAMSum example, where skipping step 2 worsens the performance due to lack of clarity in metrics
Input 1:
Eddy: Audrey dear, it’s about our meeting an Tuesday. Quite unexpectedly we’ll be having guest in on that day, some important acquaintances of Marion’s. Could we have our meeting on Monday? Otherwise on Wednesday?
Audrey: Oh I see. On Monday I’ll be busy in the gym in the afternoon and afterwards, you know, how one feels. And Wednesday we’re going with the Meadows to the Batlhus exhibition in Riehen. Should I go Friday? Or weekend?
Eddy: I’ve already seen it. It is very much worth seeing! And don’t believe the texts about teenage girls. Wait. I have to ask Marion about Friday.
Audrey: Or weekend.
Eddy: Just need confirmation. She thinks Friday will be perfect.
Audrey: The same place the same time?
Eddy: I should think so.
Audrey: Tat ta. Till Friday!
Eddy: Marion says on Fridays they’re open as early as 4 pm. So one hours earlier?
Audrey: That would be great but I might come late. Am in the office till 4.
Eddy: How long do you take to cycle to Terra del Fuego?
Audrey: 10 mins?
Eddy: There’s nothing! So see you in Terra d. F. at around 4 pm on Friday.
Audrey: Ta ta.

Output 1: Audrey, Eddy and Marion will meet at Terra del Fuego around 4 pm on Friday. Eddy has seen the Batlhus exhibition in Riehen and thinks it is worth seeing. Audrey is planning to see the Batlhus exhibition on Wednesday with the Meadows. [clarity: 5, #sentence: 3]

Input 2:
Beatric: hi Carol, did someone get a flu with fever and dry cough ?
Carol: yes, Erik and Mark were ill this week. But it wasn’t influenza, luckily
Beatric: did you give them paracetamol?
Carol: I do that every 4 hours
Beatric: and that’s all?
Carol: no you have to drink a lot, tea and honey is good
Beatric: Sarah doesn’t like honey
Carol: she doesn’t like tea either, lemon, that’s good too for the throat
Beatric: still I’m worried because the fever is high
Carol: try to alternate paracetamol and ibuprofen, every 4 hours
Beatric: are you sure I can?
Carol: yes, if she’s not better tomorrow, you may call your doc.
Beatric: that’s the point, he’s on vacation till next week.
Carol: i’m sure Sarah will recover soon
Beatric: hope so. I have a big meeting in two days in Dublin, i really have to go.
Carol: good luck. Let me know if you need some help
Beatric: sure, thanks

Output 2: Sarah has a flu with fever and dry cough, so Beatrinx asks Carol for advice on what treatment to use. Erik and Mark were also ill last week and Carol gave them paracetamol and a lot of tea and honey. Beatrinx’s doctor is on vacation till next week and Beatrinx has a big meeting in Dublin in two days. [clarity: 5, #sentence: 3]

Input 3:
Tiffany: Who’s Elon Musk?
Blair: You know nothing, John Snow.
Dale: Seriously? Never heard of him?
Tiffany: Nope, sorry.
Dale: He set up PayPal, Tesla and SpaceX.
Blair: He wants to go to the moon. He’s creating the first passenger shuttle to the moon.
Tiffany: Right? :) How much is the ticket?
Dale: Depends. Currently around $35 mil.
Tiffany: What?
Blair: Right, unfortunately, that kinda cash doesn’t fit into my piggy bank.
Dale: But the good news is that you’ll be able to watch it using VR!
Tiffany: How?!
Blair: Apparently they’re going to stream the whole thing in HD via their satellites.
Tiffany: Sounds amazing! When?
Dale: No one really knows. When they finally build the shuttle and set off, but the date is not known for now.
Tiffany: Shame. Maybe until then I’ll manage to buy myself this VR headset. :)
Blair: You really should! If not for work, you can always use it for pleasure : )
Dale: But it’s worth it for video games. You’ll get a heart attack.
Tiffany: Really that scary?
Dale: Tired it once. Nevermind the money. I’m not doing it again.
Blair: Someone got scared? ;)
Dale: But all other times are fine. And the experiences are unspeakable!
Blair: I mean, that’s what you put on the headset it’s a little awkward at first, but when you forget about it, virtual reality becomes reality.
Tiffany: Where can I buy this?
Dale: The easiest way? Online.
Blair: Remember, it’s not cheap.
Dale: <file_other> <file_other> <file_other> <file_other> There you go.
Tiffany: Cheap is not the word that best describes it.
Blair: But it’s worth it ;)
Output 3: Elon Musk set up PayPal, Tesla and SpaceX and is creating the first passenger shuttle to the moon. The tickets cost around $35 million, but the event is going to be streamed in HD via satellites. The date is yet unknown. Tiffany wants to buy a VR headset. [clarity: 5, #sentence: 3]

Input:
Morgan: Hey gorgeous, how’s your day?
Suzanne: Nothing special, it’s just one of many boring days at work. But... better now though!
Morgan: Are you working at all? 😊
Suzanne: I’m trying 😊 But you aren’t helping me, at all
Suzanne: I’m just taking a well-deserved break 😊
Morgan: I think you Suzanne
Suzanne: I miss you too Morgan
Morgan: Do you feel like going to a concert next week? Maroon 5 is playing at the Hulu Theater at Madison Square Garden.
Morgan: As it happens, I’ve got two tickets 😊 Morgan: Do you want to go?
Suzanne: Really? OMG! That’s wonderful!
Suzanne: Thank you so much!
Morgan: Oh, nothing. I just want you to be happy

Output: Output: Morgan and Suzanne discuss their day, with Morgan inviting Suzanne to a Maroon 5 concert. Suzanne excitedly accepts, and Morgan expresses his desire for her happiness. [clarity: 4, #sentence: 2]

Figure 19: Full text for an example in §2.

Reviewer paper summaries: The paper studies if task-specific demonstrations can enable a PLM to learn the underlying distribution accurately for long-form generations. They first show that LLMs don't. Next, they propose a LongGuide, an algorithm that generates guidelines as additional instructions for LLMs to achieve this. LongGuide improves open and closed-source LLMs by over 5% under zero and few-shot settings across seven tasks.
Summary of strengths: - Targets and studies a very useful task. ICL for distribution alignment in long-form generation is an important and under-studied problem.
- The method is simple and intuitive. It adds extra instructions to achieve distribution alignment.
- The paper is well-written.
- The method achieves significant improvements on benchmarks.
- Well done analysis section.
- The method is training-free.
Summary of weaknesses: The revision addresses my concerns. However, I would love to hear what the authors have said about the below weaknesses.

From the previous review:

------

- My biggest concern comes from the fact that following constraints have been shown to be a difficult task by LLMs [1]. Thus, the method becomes a bit non-relaible.
- The process is non-deterministic. I do not think the model will lead to good improvements on weaker LLMs as it depends on how well models follow constraints in instructions.
- Analysis of whether the model actually pays attention to the extra instructions has not been done. An attention-space analysis would have improved the quality of the paper.
- Does ROUGE-L capture enough information to properly evaluate the intention of the method? More analysis on if the constraints are actually followed and if following the constraints leads to improvement on benchmarks should be done.
- Related to the above, the fact that following constraints leads to improvement on tasks should be better explored.
- The primary novelty is not too novel (which is just adding constraint instructions; the automatic part is novel). Users generally add guidelines for prompting. I think the authors should focus more on their finding that "guidelines improve long-form generation."


[1] https://aclanthology.org/2023.findings-eacl.148.pdf
Comments, suggestions and typos: - What are the author's thoughts on the applicability of the method in chain-of-thought methods?
- How does the position of the extra instructions affect the performance of the proposed method? How long should the original prompt be for the model to not obey the constraint in the prompt?
- Can the authors show experiments on QA? Why wasn't this done?

This has been solved from the previous review as new baselines are added:

----- 

- Can the authors employ more baselines? What other constraint-following techniques are available? Do they improve performance?
Overall assessment scores: 4