The Ability of Large Language Models to Evaluate Constraint-satisfaction in Agent Responses to Open-ended Requests

## Abstract

Abstract Generative AI agents are often expected to respond to complex user requests that have No One Right Answer (NORA), e.g., design a vegetarian meal plan below 1800 calories . Such requests may entail a set of constraints that the agent should adhere to. To successfully develop agents for NORA scenarios, an accurate automatic evaluation framework is essential, and specifically - one capable of validating the satisfaction of constraints in the agent’s response. Recently, large language models (LLMs) have been adopted as versatile evaluators for many NORA tasks, but their ability to evaluate constraint-satisfaction in generated text remains unclear. To study this, we develop and release a novel Arithmetic Constraint-Satisfaction (ACS) benchmarking dataset. The dataset consists of complex user requests with corresponding constraints, agent responses and human labels indicating each constraint’s satisfaction level in the response. A unique property of this dataset is that validating many of its constraints requires reviewing the response as a whole (in contrast to many other benchmarks that require the validation of a single independent item). Moreover, it assesses LLMs in performing reasoning, in-context data extraction, arithmetic calculations, and counting. We then benchmark both open and proprietary LLMs on evaluating constraint-satisfaction, and show that most models still have a significant headroom for improvement, and that errors primarily stem from reasoning issues. In addition, most models exhibit a skewed constraint-satisfaction prediction pattern, with higher accuracy where the ground-truth label is satisfied . Lastly, few-shot prompting for our task proved to be rather challenging, since many of the studied models showed a degradation in performance when it was introduced.

## 1 Introduction

Generative AI agents are becoming increasingly popular, especially with the development of large language models (LLMs). As these models become more advanced and autonomous, the scope of AI agents increases, and they are now designed to include advanced capabilities and skills, such as numerical reasoning, planning, and using external tools (Pan et al., ; Di Palo et al., ; Wang et al., ; Qin et al., ). Powered by these capabilities, they are expected to handle complex user requests that may require the agent to perform multiple steps, and adhere to constraints that may be imposed by the request.
Examples for such requests include planning a trip with a given budget, creating a meal-plan with specific daily caloric-intake, or generating a fictional story with a specific number of acts and characters.
To facilitate the development of AI agents capable of addressing such complex requests, evaluating the quality of the agent’s response is essential. To illustrate this concept, the top part of Figure 1 shows an example of a complex user request to an AI agent that uses external tools, reasoning, and multi-step planning to provide an adequate response, which is finally evaluated to understand how well the response addressed the user request.

Evaluating agent responses to complex user requests is a challenging task, especially for requests that have No One Right Answer (NORA). To alleviate this, we suggest to focus on a subset of NORA requests that correspond to a Well-defined, Objective, and Verifiable (WOV) evaluation criteria, e.g., ”design a 3-day meal-plan with no meat products”. Evaluating whether the agent’s response provides 3 days of a meal plan and does not include meat is a feasible, well-defined and objective task. This is in contrast to other NORA requests that correspond to a subjective evaluation criterion, e.g., ”write a funny song about cats” (evaluation of funny is highly subjective and may depend on time, location, and culture), or requests include fuzzy or relative evaluation criteria, e.g., ”plan a short trip to Paris with a small budget”. It should be noted that some requests may only partially correspond to WOV evaluation criteria, i.e., not all parts of the request can be objectively evaluated. Nonetheless, the ability to evaluate an agent’s response with respect to only some parts of the request can be useful as well. Moreover, such requests are diverse in the sense that they span many domains and use-cases. Therefore, an evaluation framework for these cases can be very useful for developing AI agents.

To evaluate agent responses in the scenarios described above, we propose a constraint-satisfaction framework/protocol. In this protocol, a set of constraints is extracted from the user request, followed by an assessment of the constraint-satisfaction level in the agent’s response for each constraint in the set. Thus, the evaluation criterion is the alignment between the agent’s response and the constraints that are imposed by the user request. This is illustrated in Figure 1 where a user asks for a trip plan with many constraints entailed in the request. The constraints are enumerated and a set of constraints is produced. Then, the agent’s response is evaluated against each constraint iteratively and independently. A final score can then be given. Note that the constraints are formulated in natural language, and thus, the scope of evaluation is not limited by this protocol. In addition, such a protocol was previously studied and was found useful for detecting factual errors in LLM’ responses using attention patterns (Yuksekgonul et al., ) and for information-retrieval (Abdin et al., ).

Evaluating the constraint-satisfaction level in agent responses can be performed in multiple ways. One option is to utilize human raters, but this approach is often not reproducible, and more importantly, it is not scalable. Thus, an automatic evaluation framework is highly desired.
Recently, many works have utilized LLMs for various evaluation tasks, especially when the evaluation criterion is becoming more complex and intricate, such as in NORA scenarios (Chang et al., ; Li et al., ; Zheng et al., ; Liu et al., ).
For example, (Xu et al., ; Kasahara & Kawahara, ; Chan et al., ; Qin et al., ; Wang et al., ) studied LLMs as side-by-side evaluators, (Fu et al., ; Chen et al., ; Lin & Chen, ; Liu et al., ; Zhong et al., ; Chiang & Lee, ) studied LLMs for evaluating a pre-defined set of attributes (e.g., accuracy, coherence, and informativeness), and (Jiang et al., ; Min et al., ; Lu et al., ) studied more advanced evaluation protocols based on error analysis. However, they did not explicitly study the ability of LLMs in evaluating constraint-satisfaction in NORA scenarios.
In order to enable this, a specific benchmarking dataset is required.

There are many datasets that enable benchmarking LLMs on separate capabilities that are required for evaluating constraint-satisfaction. For instance, (Cobbe et al., ; Patel et al., ; Roy & Roth, ) test arithmetic reasoning, (Rajpurkar, ; Joshi et al., ) test question-answering, (Hendrycks et al., ) tests multi-level knowledge in a diverse set of fields, and (Thakur et al., ) test information-retrieval. While providing useful insights into diverse LLM capabilities, they do not test LLMs in NORA scenarios. Thus, understanding the performance of state-of-the-art LLMs in evaluating constraint-satisfaction is still somewhat limited.

To fill this gap, in this work we develop a dataset for benchmarking LLMs on the task of evaluating constraint-satisfaction in NORA scenarios. The dataset is semi-structured: each datapoint comprises a user request, a constraint, an agent response, and a binary label (annotated by human raters) for whether the constraint is satisfied, all formulated in natural language. We chose to focus on numerical constraints in order for the task to be well-defined, and since their evaluation require complex multi-step reasoning over the entire agent’s response. With this configuration, the benchmark assesses the LLM’s ability to perform multiple steps in-context to arrive at the final answer, where each step may require a different capability: reasoning, data extraction, arithmetic calculations, and counting. We thus name the benchmark ACS for Arithmetic Constraint-Satisfaction. We use the dataset to benchmark both proprietary and open popular LLMs.
The contributions of our work are as follows.

- 1.
Formulation of a constraint-satisfaction framework that facilitates automatic evaluation of agent responses to complex user-requests in NORA scenarios.
- 2.
Development and release of the ACS dataset for benchmarking auto-scorers of constraint-satisfaction. The dataset if available at blinded for submission.
- 3.
Benchmarking current state-of-the-art (SOTA) LLMs on the ACS dataset, including both proprietary models (Gemini 1.5, GPT-4o), and open models (Llama-3, Mixtral, Mistral), revealing weaknesses in some models’ capability to serve as auto-raters, as well as the challenges in effectively implementing few-shot prompting.
- 4.
Follow-up error analysis showing that reasoning is the main cause of error, and not arithmetic calculations.

Figure: Figure 1: An illustration of a complex user request to an AI agent for planning a trip with constraints. The agent should typically use reasoning, external tools and take multiple steps to provide an adequate response. Then, an evaluation process should be performed to score the quality of the response. At the bottom part, the constraint-satisfaction protocol is illustrated, where, first a set of constraints that should be satisfied in the agent’s response is enumerated from the user request. Then, the evaluation process assesses the constraint-satisfaction level in the response iteratively, for each constraint in the set.
Refer to caption: x1.png

## 2 Arithmetic Constraint-satisfaction Dataset

This section describes the ACS benchmarking dataset, including its development, properties, scope and limitations.
The aim of the ACS dataset is to benchmark LLMs on the task of evaluating constraint-satisfaction in NORA scenarios. More specifically, the dataset is focused on WOV constraints that require the LLM to perform reasoning, data extraction, arithmetic calculations, and counting. The scenarios in the dataset are taken from three domains of high interest: meal-planning, daily-schedule-planning, and workout-planning.
At a high-level, each datapoint in the dataset is structured as follows, where each item is formulated in natural language:

- •
User request: a NORA request to an AI agent that contains at least one WOV constraint.
- •
Constraint: a single constraint that corresponds to the user request that should be verified in the agent’s response.
- •
Agent response: a generated response that addresses the user request.
- •
Label: a human-annotated binary label for whether the constraint is satisfied or not.

Note that the constraints that should be verified in the response are given explicitly in the dataset, thus the scoring LLM is not required to extract these from the user request. The reason for this is to provide a common evaluation criterion to different LLMs (i.e., how well do different LLMs evaluate the constraint-satisfaction level of a specific constraint). However, future work could study and compare LLMs’ performance in evaluating the satisfaction of implicit constraints. In addition, note that each datapoint corresponds to a single constraint, even though there may be multiple constraints to each request. This structure facilitates the performance analysis of different LLMs on the constraint-level, rather then request-level.

### 2.1 Dataset Generation Process

The dataset was generated using an interleaved process of LLM prompting for generating text (user request, constraints, and agent responses), and manual modifications and filtering of the generated text (performed by humans). The latter was performed to refine the LLM output and fix inconsistencies. We used Gemini-1.0-ultra (Gemini Team Google, ) to generate the text in all of the stages describes next:

- 1.
[Manual] Crafting guidelines for how to generate user requests that entail complex arithmetic constraints. These can be thought of as seed prompts for generating the entire dataset. We used four sets of guidelines, one for each domain: meal-planning, and daily-schedule-planning, and we further separated the workout-planning domain into cardio and strength, and provided a different set of guidelines for each sub-domain. The guideline in each domain included specific constraints that should be explicitly stated in the user request. For example, in the meal-planning domain, the guideline included a caloric restriction value that should be requested, with a value taken from a reasonable pre-defined appropriate range.
- 2.
[Gemini-1.0-ultra] Generating user-requests in the three domains according to the manually crafted guidelines in the previous step.
- 3.
[Manual] Appending a final instruction to each user request that was generated in the previous step. The final instruction requested to explicitly include a breakdown of relevant numerical information, e.g., number of calories in meal-plan, working time in a daily-schedule, and exercise duration in cardio-workout-plan. This step was taken to verify that the generated responses to the user queries would explicitly address the constraints and would include numerical values that could be later evaluated by an auto-scoring system.
- 4.
[Gemini-1.0-ultra] Generating constraints for each user request that were created in step 3. In this step, Gemini was given few-shot examples in order to generate only arithmetic and counting related constraints and to control the format of the constraints.
- 5.
[Manual] Correcting the format or phrasing of the generated constraints in the previous step or adding missing constraints.
- 6.
[Gemini-1.0-ultra] Generating ”agent responses” by querying Gemini with the user requests that were created in step 3. While using Gemini-1.0-ultra, i.e., an LLM, rather than a more advanced AI agent with planning capabilities may seem inappropriate at first, recall that the aim of the ACS dataset is to benchmark LLMs on evaluating constraint-satisfaction. Thus, the agent responses in the dataset are not required to be generated by a domain-dedicated AI agent.
- 7.
[Manual] Filtering and modifications to the agent responses that were generated in the previous step, e.g., removing information from the response that may cause the constraint-satisfaction evaluation to be ambiguous, or revising the response to control whether the constraint is satisfied or not in order to diversify the data.
- 8.
[Manual] Labeling each pair of constraint and agent response as either satisfied or unsatisfied.

With the process above, 25 unique user requests were generated for each domain: meal-planning, daily-schedule-planning, workout-planning-cardio, and workout-planning-strength. Each request corresponds to multiple constraints and to a single agent response. Next, we have separated the constraints such that each datapoint in the dataset would correspond to a unique triplet of {user request, constraint, agent response}. Following all the steps above (which included manual removal and augmentation of the data) resulted in a dataset with 405 datapoints, and with a satisfied to unsatisfied datapoints ratio of 241/164 ($\approx$59% of samples are labeled as satisfied). An example for a datapoint is presented in Figure 2.

Figure: Figure 2: A datapoint example from the ACS dataset. The full agent response was trimmed for brevity. In this case, the constraint is unsatisfied since Day 1 corresponds to a total protein intake that is less then 150 grams.
Refer to caption: x2.png

### 2.2 Dataset Evaluation Scope

The agent responses in the ACS dataset may include inaccuracies and inconsistencies that are not expected to be validated by the scoring LLM, and their verification is out of scope of this work. For instance, a meal-plan may include a food item with a corresponding caloric value, but with no exact measurements. On the other hand, it may include exact measurements, but the given caloric value may be incorrect. Another example in the workout-planning domain is a cardio routine with a given value for estimated burned calories, which may be very inaccurate. Evaluating this level of accuracy in the agent response is not the aim of the ACS benchmark. Rather, the numerical values that are given in the response, which can not be further broken down into smaller components (based on the information that is given in the response), are assumed to be correct. However, any global numerical values, such as a response that states ”Here is a meal plan with at least 150 grams of protein each day..” are not assumed to be automatically correct, and should be verified by the scoring LLM. Thus, most of the datapoints in the ACS dataset can be evaluated using the following general process:

- 1.
Extract information from the agent’s response that is relevant to the current constraint, either numerical values (such as caloric values for each food item in a given day) or other textual entities (such as a list of exercises that comprise a single routine).
- 2.
Perform arithmetic operations (such as summation, multiplication, or subtraction), or counting.
- 3.
Evaluate the result with respect to the constraint.
- 4.
Potentially repeat the steps above with a different section in the agent’s response (for instance, evaluating the caloric intake of the next day in the meal plan).

### 2.3 Required Numerical Capabilities

The specific numerical capabilities that are required to evaluate each datapoint in the dataset are: counting, summation, multiplication, and date-time arithmetic. The distribution of the required capabilities in each datapoint in the ACS dataset is presented in Figure 3.
Date-time arithmetic mainly refers to the ability to understand how much time is assigned to different sections in a given schedule. Concretely, this means calculating the duration between two specific times within a given schedule, where the times are mostly expressed in ”HH:MM” format. In some datapoints, the total time should be accumulated based on multiple sections in the schedule. All the datapoints that require performing multiplication also require accumulating the results over multiple sections in the response. Thus, this capability is explicitly stated as ”Multiplication and summation” in Figure 3.

Figure: Figure 3: The distribution of the required numerical capabilities in each datapoint in the ACS dataset. Datapoints that require multiplication always require summation as well, and thus ”Multiplication and summation” is stated explicitly.
Refer to caption: x3.png

### 2.4 Dataset Properties

To make this benchmark realistic and challenging, it was designed to have several key properties. First, the relevant information that the LLM should use in its evaluation is not presented sequentially, but is rather scattered in the context window.
The maximal number of tokens from the agent response field in the ACS dataset is 1963, when calculated with Gemini 1.5 Pro tokenizer via the Gemini API (Google, ). This is far smaller than the maximal context size of current state-of-the-art LLMs, many of which support 8k to 2M input tokens (including those that will be studied in Section 3).
Another property of the benchmark is that it may contain ”distractors” for specific constraints, i.e., similar pieces of information that should be ignored since they are not relevant. Moreover, ”positive distractors” may be present, i.e., keywords that represent the constraint as being satisfied (e.g., “Here is a 2000 calorie meal plan” when the constraint is “the meal-plan should be up to 2000 calories a day” and the label is “unsatisfied”).
Another challenging property of our benchmark is that in order to fully verify some constraints, the LLM should perform an iterative evaluation process. This refers to performing multiple independent instances of the same evaluation process using different pieces of information from the context. An example for this is when evaluating the daily caloric intake of a multi-day meal plan, the calories for each day should be calculated independently.
Thus, unlike benchmarks that rely on simple keyword matching or isolated text snippets (such as in Question-Answering, NLI, and sentiment analysis), the ACS dataset demands a comprehensive/holistic evaluation of the agent response.
Lastly, the benchmark does not require any domain-specific or specialized knowledge. The complexity of evaluating each datapoint in the dataset can be considered to be at an elementary-school level. This is important in order to fairly evaluate fundamental LLM capabilities that are desirable across domains, without giving an advantage to domain-specific LLMs. We believe that these properties make the ACS benchmark useful in assessing the capabilities that are crucial to successfully incorporate LLMs into a wide range of applications.

## 3 Experiments

The ACS dataset was used to benchmark several LLMs on the task of evaluating arithmetic constraint-satisfaction in NORA scenarios. Recall that the ACS dataset contains a ground-truth binary label for whether the constraint is satisfied in the agent’s response. Thus, in this study, the LLMs were instructed to evaluate the agent’s response with respect to the constraint, and were instructed to provide a final yes/no decision for whether the constraint is satisfied.

### 3.1 Setup

The studied LLMs include Gemini 1.5 Pro (version 0514), Gemini 1.5 Flash (version 0514), Gemini 1.0 pro (stable version 002) (Gemini Team Google, ), GPT-4o (version 2024-05-13) (OpenAI, ), Llama-3-70b-chat, Llama-3-8b-chat (AI@Meta, ), Mixtral-8x7b-instruct-v0.1 (Jiang et al., ), and Mistral-7b-instruct-v0.2 (Mistral AI, ).
Gemini models were accessed through the Gemini API, OpenAI’s GPT-4o was accessed through the OpenAI API, and the open models were deployed to a Vertex AI endpoint.
Default text-generation parameters were used for each LLM. Each LLM was then used to evaluate the entire ACS dataset. The LLMs were instructed to use a chain-of-thought (CoT) reasoning process (Wei et al., ) and perform any necessary calculations explicitly, rather than relying on final values stated in the given plan. Furthermore, two prompting configurations were studied: zero-shot and few-shot with two evaluation examples taken from a trip-plan scenario (which is out-of-domain).
The first example contains a 3-day itinerary including prices for each element and a daily-budget constraint. The second contains a driving plan of multiple segments with driving time and average speed for each segment and a constraint of maximal driving distance per segment. The evaluation process in the example shows the model how to extract the relevant information (item prices, or driving time and average speed), perform calculations (summation and multiplication), and compare the result against the constraint to decide whether it is satisfied. The evaluation prompt is presented in the appendix in section A.1, and the few-shot examples are presented in section A.2. Then, each LLM’s final decision, i.e., prediction, regarding the constraint-satisfaction in the agent’s response was extracted from the full LLM evaluation response using regex, and was used to analyze the performance of the LLM.

### 3.2 Results - accuracy metrics

Following the predictions that were made by each LLM, the following accuracy metrics were calculated: overall-accuracy, which is the constraint-level prediction accuracy, and $F_{1}$ score of predicting each of the labels: “satisfied” and “unsatisfied”. By this separation, we can study whether an LLM has a bias towards predicting one label over the other. The accuracy metrics are presented in Table 1 for all studied LLMs. As can be seen, GPT-4o achieves the best accuracy scores in both zero-shot and 2-shot configurations. The high accuracy score of 97.04% shows that GPT-4o can serve as a rather reliable auto-scorer for the kind of tasks presented in the ACS benchmark. The next best performing model is Llama-3-70b-chat at a zero-shot configuration, but it performs significantly worse, at an accuracy rate of 90.62%. The performance of the remaining models is even worse, indicating their lack of competence in performing as reliable auto-scorers in the constraint-satisfaction task studied here.

Another insight derived from Table 1 is the difference in $F_{1}$ scores between constraints with positive (satisfied) and negative (unsatisfied) labels. All models but GPT-4o seem to predict positive datapoints much more accurately than negative datapoints. This phenomenon may be attributed to the fact that the ACS dataset contains “positive” distractors (see subsection 2.4), i.e., keywords in the agent response that imply that the constraint is satisfied (e.g., “Here is a 2000 calorie meal plan” when the constraint is “the meal-plan should be up to 2000 calories a day” and the label is “unsatisfied”). If these “positive” distractors are indeed the causes for the imbalance between the labeling classes, it may show a weak-point of the models in performing similar tasks objectively. However, this is just a hypothesis at this stage, and further analysis is required to validate it, and is suggested for future work.

Finally, when comparing the performance of the models in the zero-shot versus 2-shot configuration as seen in Table 1, an interesting behavior is observed. Some models present an improved performance with respect to the $F_{1}$ score in the 2-shot configuration, such as Gemini 1.5 Flash (increase of 4.44 percentage points), Gemini 1.0 Pro (increase of 3.46 percentage points), and perhaps Gemini 1.5 Pro although the difference is not major (increase of 0.98 percentage points). This shows that these models can benefit from few-shot prompting strategies since they can guide the models to evaluate constraint-satisfaction more accurately.
In contrast, the accuracy of the open models decrease in the 2-shot configuration, compared to zero-shot. This is most noticeable in Mixtral-8x7b-instruct-v0.1 (decrease of 7.16 percentage points) and Llama-3-8b-chat (decrease of 4.44 percentage points), while the remaining models correspond to a decrease of less than 2 percentage points. The cause for this decrease in performance is not clear from this analysis alone, but a potential cause may be the out-of-domain examples.
Lastly, GPT-4o achieves a high level of accuracy in both zero-shot and 2-shot configurations, which is a desirable behavior that suggests increased generalization capabilities, compared to the remaining models.

Table: Table 1: Overall-accuracy, satisfied $F_{1}$, and unsatisfied $F_{1}$ scores achieved by each LLM in evaluating the constraint-satisfaction level of the ACS dataset in zero-shot and 2-shot configurations. The best-performing results are highlighted in bold.

### 3.3 Results - error analysis

To further analyze some of the LLMs’ performance and gain insights into their capabilities, an error analysis of selected models was performed in the 2-shot configuration. The inspected models are Gemini 1.5 pro, Gemini 1.5 flash, GPT-4o, and Llama-3-70b-chat.
Since the evaluation prompt template invokes CoT reasoning (see Appendix A.1), it enables examining the full evaluation process of each LLM and identifying the cause of error in cases of incorrect final prediction. The errors were manually analyzed and categorized into the following buckets:

- 1.
Reasoning: which was further divided into these subcategories
(a)
Extraction: failing to extract all relevant items, or extracting additional irrelevant items
(b)
Counting: extracting a correct list of items to count but the final value is wrong
(c)
Schedule understanding: incorrect deductions from a typical schedule structure
(d)
Other: intermediate or final reasoning steps with logical errors
- 2.
Calculation: errors in summation, multiplication, or date-time related errors (mainly time calculation “subtraction” errors)

Table 2 shows the number of times each error category occurred for incorrectly predicted data points. The percentage of each error category relative to the total number of errors is shown in brackets.
Note that for all LLMs, most errors are caused by erroneous reasoning steps. This highlights the fact that incorporating tool-use for arithmetic calculations is not expected to be the most important step for improving the performance of these models in similar tasks. In addition, GPT-4o did not make calculation errors, but both Gemini models and Llama-3-70b-chat made such errors with relatively similar proportions. Moreover, it seems that correctly extracting in-context relevant information is more challenging for Gemini 1.5 flash, compared to the other LLMs. For Llama-3-70b-chat, counting seems to be more challenging compared to the other models. Finally, recall that a model might make any error and still predict the final label correctly. Thus, this analysis has limitations and should not be interpreted as showing that any LLM was immune to making specific errors.

Table: Table 2: Error analysis counts of some of the studied LLMs. Absolute counts are shown and their portion from the total number of errors is in brackets. Most errors are caused by incorrect reasoning steps.

## 4 Limitations

The work presented here offers useful insights in to the ability of LLM to serve as auto-scorers for the task of constraint-satisfaction in NORA scenarios, but it has some limitations. First, the ACS dataset has a limited scope and size. It spans three main planning domains, which are useful, but represent only a small fraction of real-life use-cases. The set of capabilities that is required to correctly evaluate the dataset is also limited, as described in section 2, and thus it does not reflect the full set of capabilities that are required from auto-scorers to evaluate constraint-satisfaction. In addition, the size of the data is limited to 405 datapoints, where each corresponds to less than  2000 tokens. With the increased interest in very large context sizes (Lin et al., ; Song et al., ; Ding et al., ; Gemini Team Google, ), it may be very useful to study the ability of LLMs to evaluate constraint-satisfaction with very large context sizes. Currently, the ACS dataset does not include datapoints that are composed of very large tokens ($>10k$), but this is suggested for future work. Finally, while GPT-4o achieves very high accuracy scores when benchmarked against the ACS dataset, the remaining LLMs, especially the open models, have a significant headroom for improvement. Thus, the ACS dataset can be useful for the development of more advanced LLM-based auto-scorers.

Next, the experimental study in section 3 also has some limitations. Recall that the study in section 3.2 measures the accuracy of the LLMs in predicting the correct constraint-satisfaction label - either satisfied or unsatisfied. With the reduction of the evaluation task to binary prediction, it is possible for the LLM to make some errors, whether in the data-extraction, reasoning, calculation, or counting step, but nonetheless predict the correct class, assuming such potential errors are insignificant. As an example, consider the constraint: “the meal-plan should be at least 1700 calories” that corresponds to a meal-plan with 1800 calories. An LLM that during the evaluation calculates either 1800 (correct) or 2000 (incorrect) calories, could predict the same correct label: satisfied. This paradigm facilitates the analysis of the LLM accuracy (no additional steps are required to extract intermediate numerical values that the LLM is expected to produce) but it may obscure the LLM actual performance to some extent. For future work, additional experiments could be performed that test the LLMs accuracy in more detail, for instance, by examining the accuracy of the numerical values that the LLMs produce during their evaluation. In the example above, an additional step that extracts the meal-plan calories that were explicitly calculated by the LLM and verifies this value against 1800 could be very insightful.

Finally, the study in section 3.3 presents an error analysis with specific error categories, which are just a single way to cluster the error “buckets”. Furthermore, the LLM could potentially make multiple errors corresponding to multiple categories, but we chose a single category that best describes the most significant error in the LLM output. Thus, this is a subjective and not an exhaustive analysis. We leave a more detailed analysis for future work.

## 5 Conclusions

This work presented a novel dataset for benchmarking auto-scorers using a constraint-satisfaction framework. The experiment results showed that the task of evaluating agent responses with respect to constraints that require performing in-context data extraction, reasoning, and elementary-school level arithmetic calculations and counting is still challenging for many state-of-the-art LLMs. GPT-4o was the only model that achieved satisfactory accuracy scores, among all the tested models. In addition, ”positive distractors” in the agent’s response, i.e., keywords that represent the constraint as being satisfied even though this may not be the case, may pose a challenge to LLMs that aim to score the response. Regarding the prompting strategy, not all models may benefit from few-shot prompting for the task studied by the ACS dataset. Moreover, it may be detrimental to their performance, and thus, this prompting technique should be handled with care when designing an LLM-based auto-scorer. Finally, the primary source of errors is identified to be due to reasoning rather than arithmetic calculation issues, suggesting that incorporating external tools for calculation purposes may not lead to significant performance improvements.

## Appendix A Appendix

### A.1 Evaluation prompts

The evaluation prompt that was used for all the LLMs in the experiments is shown below in Jinja syntax Pallets ():

### A.2 Example given to the LLMs for how to evaluate an agent’s response

The trip-planning examples that were given to the LLMs in the 2-shot prompting configuration for how to evaluate an agent’s response are shown below:

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: - It would be interesting to verify if implicit constraints are satisfied. For example in the event planning scenario, ensuring that the agent doesn't schedule an event in two different places at the same time. Could you comment a bit on how the ACS dataset could be extended to address these scenarios.
soundness: 2
strengths: - Evaluating if the LLM's responses satisfy the user specified constraints is important in many realistic deployments of LLLM based AI agents. Therefore, developing a benchmark dataset for these kinds of situations is definitely called for.

- The quality of the presentation is overall quite good and therefore the paper is relatively easy to follow even by readers outside this research area. Most of the technical details presented in the paper are discussed in a relatively clear manner.
summary: The paper presents an evaluation framework for generative AI agents (LLM based agents) for No-One-Right-Answer (NORA) scenarios. Specifically, the focus of the paper is on developing a benchmark dataset called Arithmetic Constraint Satisfaction (ACS) dataset for evaluating if certain arithmetic constraints are satisfied by the agent's response. In particular, verifying that such a constraint is satisfied requires basic operations such as summation, subtraction, multiplication and counting, as well as the entire generated response instead of just specific parts of it. The empirical evaluation is carried out using both closed as well as open models, including OpenAI's GPT models, Google's Gemini models as well as Meta's Llama models, respectively. The results demonstrate the capability of modern language models to generate NORA responses that indeed satisfy the user specified constraints.
weaknesses: - Most modern LLMs do fairly well on arithmetic reasoning problems and the results in table 1 aren't actually surprising. Therefore, I think the proposed ACS dataset may be too easy for constraint satisfaction task considered.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 1
questions: 1. Who were your human raters/annotators? I'd appreciate more detail on that aspect of dataset construction (and how/whether these annotations were validated e.g. via inter-annotator agreements).
1. Could you include examples of the actual prompts across all four domains? In particular, is the user query included in these? It doesn't seem to be in the (out of domain) few-shot examples shown in the appendix.
1. In step 6 of section 2.1, you mention the possibility of using other models to generate responses for this dataset. Did you compare generations across models? I'd be especially curious to know if some are more or less like real human requests or if they are easier or harder for different models to evaluate.
1. Could you provide examples of each error type? I didn't see them in the main text or appendix, making it hard to understand what exactly the error types mean.
1. Line 74: What do you mean when you say that using human raters often isn't reproducible? I would have assumed that a failure to reproduce human ratings with other humans would mean that there is disagreement about the ground truth of the claim (and thus the constraint wasn't well-defined or objective), but I think I must be misunderstanding what the sentence is trying to say.
soundness: 2
strengths: 1. The authors restrict their analysis to arithmetical conststraint evaluation on problems constructed to have a well-defined ground truth, sidestepping a significant problem of some previous benchmarks.
1. The paper evaluates both proprietary and open models on the dataset, providing broad coverage of different LLM families.
1. The authors provide analysis beyond just raw accuracy scores (F1, error categorization) and show performance on more than one prompt type. 
1. The authors will release the dataset after the review process, which is good practice.
1. Section 2.1 provides a thorough overview of the steps involved in the dataset generation process.
summary: The authors present a 405 item benchmark meant to analyze the ability of LLMs to evaluate whether a numerical constraint has been satisfied in a given input. The authors generate input texts and numerical constraints using a series of LLM calls together with manual cleanup: one to generate a constraint-including request, another to generate a responses to these requests, and another to extract numerical constraints from the LLM-generated request.    
Final input texts come in four varieties: meal plans, daily schedule plans, cardio workout plans, and strength workout plans. Each prompt presents the input text without any further context, and then asks the LLM to evaluate whether one specific constraint is satisfied or not. 

The authors then evaluate models from the Gemini, GPT, Llama, Mixtral, and Mistral families on their benchmarks, finding that performance varies. All models score over 72%, and GPT-4o achieves an accuracy of 97% on the dataset. For a subset of these LLMs, the authors then manually categorize the errors seen in their CoT traces. The authors also propose using a variation of their benchmark generation process as a protocol for automatic evaluation of LLM responses to user queries.
weaknesses: In general, this paper would benefit from substantially more time, a clearer focus (is it aiming to be a benchmark or a validation framework?), and a more carefully constructed and relevant dataset.
In its current form, it feels far from complete, and I cannot recommend it for publication. I'll do my best to constructively break down why I believe this:

The premise of the paper is valid and asks an important question: can LLMs be useful as automatic constraint satisfaction verifiers for LLM responses to user queries?
The current direction has promise for making a useful contribution to answering this question, but only if much more is added.
1. The paper raises many immediately relevant questions but dismisses them as further work throughout the body of the work (lines 158, 352, 433, 447, 455). This "further work" includes significant amounts of analysis that I believe should actually be in the main results of this paper, especially the F1 score analysis (line 158) and examining intermediate numerical values rather than just final boolean evaluations (line 447). 
1. GPT-4o succeeds on over 97% of all problems in the dataset, thus saturating it. If this were a benchmark which was a well-validated representation of the domain it claimed to examine, this could be taken as evidence that this model is already sufficient for the task of evaluating constraint satisfaction in responses to user queries. However, as the authors themselves admit, this is a deliberately simplified dataset: only one constraint is evaluated at a time, and only the simplest kinds of arithmetic constraints are represented. A more useful evaluation set would either provide significant evidence that the current state of the art is sufficient for the task or would highlight where and how SoTA currently fails, thus giving potential directions for improvement (whether train time or via inference-time augmentations).
1. The error analysis is unclear and incomplete. The biggest category by far is "other." The classification of errors is confusing: counting is considered a "reasoning" task, while multiplication (a procedure that, in a sense, generalizes counting) isn't. In general, the definition of reasoning here seems both very broad and inconsistent. Why is information extraction a reasoning task? For clarity, I would recommend removing the "reasoning" label entirely from Table 2 and its associated analysis paragraphs. 
1. As the authors mention in the introduction, there have been many prior benchmarks and analyses that have looked at the same LLM abilities as those being tested here. The fact that the generation being evaluated by the LLM was in response to a "NORA" prompt doesn't fundamentally change the nature of the benchmark questions, as all of them *do* have one right answer (either "satisfiable" or "unsatisfiable"). This is especially true given that the full dataset was automatically generated rather than sourced from a real setting.
1. The dataset presented is too small and irrelevant. While the authors claim that it is "difficult" and "realistic," it is (as mentioned) already saturated, and no justification is provided for its realism. Currently it consists of a handful of very small artificial domains and uses an LLM as a user proxy but does not attempt to examine the validity of this use. A much more useful dataset would consist of actual user requests "in the wild" together with a selection of LLM generations (ideally across various models). At the very least, any benchmark purporting to answer the questions this paper is interested in should validate whether its benchmark is reflective of the sorts of queries and constraints LLMs are likely to encounter in practice.
1. The paper's focus is also unclear. The introduction mentions an automatic evaluation framework for LLMs tasked with constrained query generation and presents a figure, but this framework is not mentioned in the body of the paper. Multiple pieces of this system are not described in detail: What is the scoring system? What is the intended role of this system? For example, is the generation model reprompted if the evaluation comes up negative (or just has too low a score? in which case how is the cutoff calculated?)? If the paper is meant to be a benchmark paper, then it is unclear what the purpose of describing the framework is, unless it is modified and expanded on to create something like a living benchmark which grows with more queries or similar. If the framework is meant as a standalone contribution, then the paper must include not only further description but also further analysis of it, as well as evaluations of whether and how much it improves performance.

## Nitpicks/Typos
1. The paper introduces a number of new (nonstandard) acronyms. In particular, WOV is used only three more times after it is introduced. This makes the paper harder to read. It would be clearer if these terms were spelled out each time.
1. Line 52: while the proposed benchmark does seem to have well-defined constraints (I haven't seen the prompts), the use case it purports to model often will not. Non-arithmetical constraints which may seem at first blush to be objective and well-defined may not fully be. The example the authors give "a 3-day meal-plan with no meat products" arguably has this property. Must a 3-day meal plan include three meals a day? Does fasting for three days satisfy the requirements? Which definition of meat are we using? Does fish count? Do insects? Does lab-grown meat count? E.g. the state of Missouri required "meat" to be derived from "harvested production livestock or poultry," which would answer no to all three of the meat questions. In general, polysemanticity and linguistic ambiguity makes it very difficult to make strong claims about whether constraints are truly well-defined. If the authors wish to retain this aspect of their analysis, I think it is important that they discuss these issues, and whether and to what extent well-defined, objective, and verifiable criteria exist beyond mathematical tasks.
1. Line 105: "The dataset **if** available at *blinded for submission*." (Typo)
1. Line 167-168: "generate the text in all stages decribes next:" (Typo)
1. Line 298: The authors claim that the problems "can be seen" as "elementary school level," but provide no justification for why this is true or why it matters. It is well-known that LLMs fail on simple grade school tasks like multiplication (even with CoT/scratch space), but can score highly on some graduate level exams. Evaluations and rankings based on approximately equivalent human school level do not seem particularly meaningful for these systems.
1. Line 432: "composed of very large tokens" should be "composed of very large numbers of tokens"

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: The authors could consider adding logical constraints, such as Boolean constraints, which could incorporate both numerical and logical elements.

The writing is sometimes hard to follow. The authors could provide more examples for each component. Initially, I thought the paper focused on a dataset intended for LLMs to generate correct responses for questions with constraints. However, midway through, I realized the LLMs are actually expected to assess whether the provided responses satisfy the constraints. I hope the authors can clarify this at the beginning of the paper.

- Line 105: “The dataset if available” should be "is" instead of "if".

- Line 162: This section could benefit from examples at each step.

- Line 242: There are two periods.

- Lines 268-269: It is unclear how "The maximal number of tokens from the agent response field in the ACS dataset is 1963" relates to "relevant information is scattered in the context window."

- Lines 287-289: The meaning of "keywords that represent the constraint as being satisfied" is unclear, as is the example that follows. And why is “the label unsatisfied” in the example?

There are too many uses of right” , In LaTeX, you can use `' instead.
soundness: 2
strengths: Strengths:
1. Development of the ACS dataset for benchmarking auto-scorers of constraint satisfaction.
2. Benchmark testing on various LLM models.
summary: This paper introduces a new dataset for QA questions with constraints. The dataset comprises four parts: user question, constraint, agent response, and the correct label for the response. It focuses on tasks related to arithmetic calculation and counting. The paper also evaluates several well-known LLMs on the new dataset and finds that GPT-4o achieves the best performance. The results suggest that the primary source of errors stems from reasoning challenges rather than arithmetic calculation abilities.
weaknesses: Weaknesses:
1. Some parts of the writing are unclear.
2. There are too many manual steps involved in dataset generation.
3. The dataset is relatively small.

---

confidence: 3
contribution: 1
initial_rating: 3
presentation: 4
questions: Two high-level questions; 
1) What was the motivation behind studying these particular domains: meal planning, daily scheduling, and workout planning?
2) As a takeaway from this study, would you suggest that we need datasets for domain/application to advance LLMs constraint satisfaction evaluation?
soundness: 3
strengths: First of all, I enjoyed reading this paper. The paper write-up is easy to follow, and the material is presented naturally with suitable examples in the right places. Thanks to our authors!
A non-trivial amount of work is needed to compile this dataset, which is said to have contributed to the community and can be extremely valuable. 
The authors are honest about the design choices that went into this process, clear about the dataset created and utilized, and, in particular, cover several limitations of their work. The paper attains a scholarly approach.
summary: This paper studies an LLM evaluation setup called NORA (no one right answer). 

Within this context, it proposes a dataset and the associated design decisions to evaluate the LLM response generation for queries in different domains, including meal planning, daily scheduling, and workout planning. 

Several LLMs and prompting techniques are then evaluated in this dataset, which is contributed to the community.
weaknesses: I am following the introduction of the dataset, which is a non-trivial amount of work and nicely contributes to the literature and community. This is all great. 
What is less clear to me is 1) the high-level motivation behind doing this, and 2) the ultimate path this might take our understanding further. 

Specific to the evaluation dataset, as noted by the authors, GPT4o achieves %97+ accuracy. It is not clear how much further this can be improved (I assume there will always be some data uncertainty etc.) and what value remains. This is quite important if this dataset is to be serving as a "benchmark". This might not be the case here.  
I understand that experiments rely on LLMs only but I would have liked to see a comparison with an LLM with tools access as another additional baseline. Some of the steps required such as counting, arithmetic, summation, etc. are clear candidates for tool usages. The paper claims that most errors are due to reasoning rather than calculation (which I tend to agree) but still an LLM with a tool is an immediate baseline to include in experiments. I wonder if that would open-source LLMs, which are trailing behind GPT, to close the gap.

Meta-review: Reviewers agreed that datasets and evaluations of LLMs are important, yet also raised various concerns on the approaches of this paper, including conducting more immediately relevant questions, providing more complete and clear analysis, and improving the datasets in quantity and quality. The authors did not provide a response so the scores and sentiments remained negative after the rebuttal phase.

Decision: Reject