WIQOR: A dataset for what-if analysis of Operations Research problems

Authors: Aditya Parashar, Natalia Kosilova, Ari Kobren
Primary area: datasets and benchmarks
Keywords: datasets, operations research, counterfactual reasoning, mathematical reasoning

Abstract:
We formalize the mathematical program modification (MPM) task, in which the goal is to revise a mathematical program according to an inquiry expressed in natural language. These inquiries, which we refer to as what-if questions, express a desire to understand how the optimal solution to an optimization problem changes with the addition, deletion or revision of constraints. In detail, each MPM instance
is a triple consisting of: 1) a natural language specification that summarizes an optimization problem, 2) the canonical formulation of the problem, and 3) a natural language what-if question. The goal is to predict the updated canonical formulation with respect to the question. To support the study of this task, we construct WIQOR, a dataset of 1,946 MPM instances, derived from NL4OPT (Ramamonjison et al., 2023), but with the number of decision variables extended to more than 30 for some problems. In experiments, we observe that Llama 3.1 70B instruct under the in-context learning paradigm achieves 69% accuracy on the easiest test instances, but only 36% accuracy on the most complicated problems. We release WIQOR in the hopes of spurring additional study of MPM and ultimately enabling non-technical users to conduct what-if analyses without the help of technical experts.

Decision: reject
Reviewer scores: 6, 3, 5

Reviews and discussion:
<thread>
creation date:
1730245934215

thread_type:
official_review

<thread_object>
creator: reviewer_mpi3

<content>
summary:
- This paper proposed a new task, mathematical program modification (MPM), whose goal is to revise a math program (MP) based on a natural language instruction.
- The paper presented a dataset WIQOR, of 1946 MPM instances, derived from NL4OPT dataset. WIQOR contains 4 set, train, dev, test-base and a harder test-VarAug. To build the dataset, the authors first modify the original MP with 4 potential ways (LC, CC, CDR, VIM, explained in paper), and then prompt LLMs, as a reverse engineering approach, to generate the **What-if questions** based on the difference between the original MP and the modified MP.
- The author test 3 variants of Llama model on the proposed dataset and the experiment results show it is a challenging dataset.

strengths:
- The collected dataset could be a contribution to the community, however if it is actually emulating the real-life scenario is still questionable.
- It is interesting to see how different constraint types and what-if question types pose different levels of challenges for LLMs.

weaknesses:
- This paper is clearly a rushed work. The presentation is draft-like ( For example, the margin of Figure 6, the unfinished appendix A1,2,3), the writing is a bit verbose ( for example Section 2 related work and Section 3.1 & 3.2).

- The evaluation section is limited. 
    - As shown in Table 3, the dataset is only evaluated on very few models, Llama-3.1-8B/70B and a code-Llama. 
    - When comparing ICL variant, zero-shot performance is not tested. 
    - When comparing CoT performance, ablation study should be conducted to compare the performance with and without CoT.
    - As very few models are evaluated, the conclusions drawn in Section 6 is not well supported.

- The motivation for the proposed dataset is not convincing enough, and the scope of the project is relatively small.
    - The whole motivation of the dataset is built on the assumption that in real-life people will communicate/modify a MP in such a way, but your specification S is just summary and do not contain detailed information. Why do you think that's the actual way people communicate?
    - The proposed modification approach should be compared with a NL-to-MP baseline, where the original optimization text + modifications is given to the LLM to generate a new MP. If this actually works better than modifying the existing MP, I don't see the motivation for this approach.
    - How good is the coverages of What-if questions? Do they cover all potential aspects of a MP problem? Are all the MP problems linear programming problems? You can include a limitation section to discuss the coverage of the What-if questions.

questions:
- Are all the MP questions included Linear programming? If not, did you treat linear and non-linear differently or did you investigate their performance difference?
- What's the actual prompt you used to evaluate the performance of Llama-3.1? Did you include specification in your prompt? Please show a template example.
- You specification S does not contain specific description about the variables. Does it mean after the modification, the specifications are still guaranteed to be valid?
- What's the motivation for rewriting the optimization problem to the specifications?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730479833308

thread_type:
official_review

<thread_object>
creator: reviewer_6scn

<content>
summary:
This paper develops a novel dataset, using the dataset from NL4OPT as a seed to expand the original dataset. The complexity of generated problems, as well as the number of variables and constraints, is gradually increased to obtain more complex data. The provided canonical form is correctly modified in response to the what-if question.

strengths:
- The paper proposes a self-instruction-based approach to synthesize complex reasoning data, using a progressive method to generate more complex problems and solutions.

- The article is well-written, and the method for expanding the dataset is simple and clear.

- It explores some interesting combinations of conditions and variables, which is very helpful for investigating the boundaries of linear programming problems.

weaknesses:
- The novelty of the data generation method is limited[1,2]. I believe that the self-instruction-based method may face serious mode collapse[3] issues, and it’s necessary to discuss the presence of this problem in mathematical problem synthesis. As for improvements, I think a human-in-the-loop approach could be provided for optimization.
- Using exact match to calculate accuracy is unreasonable; why not provide results on the solver instead? Exact match can be overly restrictive and often misleading in evaluating mathematical and reasoning tasks, as it treats answers as binary (correct or incorrect) without considering near-correct solutions that may only have minor calculation differences. Real-world problem-solving often requires that a solution meet certain functional criteria, like feasibility within constraints or optimality, rather than matching a single predetermined answer exactly. Using solver-based evaluations provides a nuanced assessment, confirming if solutions meet problem requirements under real conditions. Solvers can verify the feasibility and robustness of a solution within the problem’s constraints, accounting for slight variances and rounding errors that exact match might unfairly penalize. This approach aligns the evaluation with real-world performance, offering clearer insights into the model's practical applicability and helping to identify areas for meaningful improvement. By assessing solutions based on their functionality and effectiveness, solver-based evaluation ultimately yields a more refined measure of a model's problem-solving capability, moving beyond rigid surface-level accuracy.
- Can it be extended to more complex mixed-integer programming problems? Mixed-integer programming introduces additional complexities compared to standard linear programming due to the presence of integer constraints, which often lead to a more challenging solution space and can significantly increase the computational difficulty of finding optimal solutions. For example, extending this method to MIP might require adjustments in handling discrete variables, which can limit certain relaxation techniques commonly applied in continuous optimization problems. Additionally, it would be important to investigate how the current framework could address or adapt to the combinatorial nature of integer constraints. This could involve rethinking the approach to feasible region exploration, as well as adapting constraint handling mechanisms to account for mixed integer domains. Overall, this extension could open up new application avenues, and exploring the specific requirements for such an adaptation might reveal valuable insights into the method's versatility and practical scalability across diverse optimization scenarios.
- I haven't seen any code or datasets; I would like to carefully check whether the data questions truly align with real-world solving objectives. 
- I believe it is essential to provide the solvability rate of the generated problems and to include test results across different solvers, comparing their performance.

**Reference**  
[1] Self-Instruct: Aligning Language Models with Self-Generated Instructions.  
[2] Wizardlm: Empowering large language models to follow complex instructions.  
[3] Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data

questions:
- I believe many experimental results are missing. For instance, have you evaluated whether the generated solution formulas are solvable? Additionally, numerous generated problems need to be tested on the solver.
- How is data quality ensured, given that many problems generated by the model often do not exist in the real world? Are there any automatic checks or manual evaluation steps to filter out unrealistic problems? I believe that the validity of the generated problems needs to be supported by detailed experimental results to be convincing.
- How is data diversity ensured, considering that simple semantic similarity is typically not a good metric for mathematical reasoning tasks?  I believe it is necessary to provide some metrics or analysis to demonstrate the diversity of the dataset and to ensure that mode collapse has not occurred.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730608671582

thread_type:
official_review

<thread_object>
creator: reviewer_fsj1

<content>
summary:
This paper introduces a new task called Mathematical Program Modification (MPM), where a mathematical program is revised in response to a natural language inquiry, referred to as "what-if" questions. The authors also examine how problem complexity, specifically the number of variables, impacts model performance. Experiments with the Llama-3.1 series show that more challenging problems remain difficult for large language models, indicating significant room for improvement in handling complex modifications.

strengths:
- The task is novel, challenging, and highly relevant for real-world applications.
- The dataset is thoughtfully designed, with components like problem specifications that enhance clarity and usability.
- The task is easily scalable; complexity can be adjusted by increasing the number of variables, as demonstrated in this work.
- The task can be effectively generated and tackled by large language models like LLaMA, achieving promising accuracy levels.

weaknesses:
My main concern lies in the evaluation aspect:
- The paper evaluates only three models from the LLaMA series. As a benchmark, this is somewhat limited. Could the authors include evaluations from more advanced models, such as GPT-4 or Claude-3.5? This would give the community a clearer understanding of model performance on this task.
- The what-if questions are generated by LLaMA models and then evaluated by similar models, which may introduce potential biases. Including results from a broader range of models would help clarify whether such biases are present.
- The paper could benefit from additional analyses, such as self-consistency performance and pass@k accuracy. These insights could add further depth to the evaluation.

questions:
See weakness.
And the right part in figure1. is vague.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733273700778

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A summary of the discussion phase for the perusal of the PCs and ACs

comment:
## Strengths
All reviewers agreed that the work presents a novel and valuable contribution to the field, with particular praise for the WIQOR dataset. They highlighted the dataset’s relevance for evaluating large language models (LLMs) in the context of mathematical program modification (MPM), noting its potential to address real-world challenges in optimization. Reviewers appreciated the thoughtful design of the dataset, including its scalability and the use of human evaluation and heuristics to ensure data quality. They also acknowledged that the evaluations underscored the difficulty of the task and demonstrated the potential of LLMs in handling complex optimization problems. Overall, the work was recognized as a significant and original contribution to the field, offering a challenging task for current LLMs with broad implications for optimization and natural language processing.

## Weaknesses, questions and their Rebuttals
### Reviewer fsJ1:

- Reviewer pointed out the limited evaluation with only three LLaMA models, so we we expanded our experiments to include GPT-4 in zero-shot, few-shot, and CoT settings. The results showed that GPT-4 demonstrated superior performance compared to LLaMA models, supporting our conclusions.  

- Reviewer raised concerns about potential bias from using the same model for generating and evaluating the "what-if" questions. We clarified that the questions are deterministic and based on algorithmic modifications, minimizing bias. We validated this by including GPT-4 evaluations, which showed similar trends, confirming the fairness of the evaluation.

- Reviewer suggested including additional evaluation metrics like pass@k accuracy and self-consistency performance. We tested Llama 3.1 70B Instruct on the task with pass@5 and presented the results although further clarified that exact-match accuracy is more suitable for this deterministic task, as it directly measures the correctness of the formulation modifications. Pass@k is more appropriate for tasks with multiple valid solutions.

- Reviewer asked for clarification on Figure 1. We clarified that the right side of Figure 1 shows the modified constraint, with the ellipsis indicating unchanged variables.

**Outcome**: Reviewer was satisfied with the rebuttal and increased their score to 6.

---

### Reviewer 6sCn:

- Reviewer mentioned the limited novelty in the data generation method and mode collapse concerns. We clarified that novelty of the data generation method was not the goal of our work, the method uses heuristics and human evaluation to ensure high-quality data. Mode collapse was assessed using the Self-BLEU score (0.50156), showing enough diversity in the generated questions.  

- Reviewer expressed concerns about the exact-match accuracy being too restrictive and suggested solver-based evaluation. We clarified that exact-match accuracy directly measures the correctness of modifications, making it the most appropriate metric for this task. Solver-based evaluation is more suited for solution feasibility, which is not the focus here.

- Reviewer asked about the lack of extension to mixed-integer programming (MIP). We clarified that our current work focuses on linear programming, but our methods can be adapted for MIP in future research. 

- Reviewer questioned the realism of the dataset. We clarified that the WIQOR dataset, based on NL4Opt, tests reasoning in practical optimization tasks. While not real-world problems, they are designed to assess model performance in structured problem-solving tasks.

**Outcome**: Reviewer didn't engage further and did not change their score.

---

### Reviewer mpi3:

- Reviewer pointed out draft-like presentation and verbosity. We clarified that we improved the paper’s formatting (e.g., Figure 6) and made Sections 2 and 3.1–3.2 more concise, focusing on key concepts while removing unnecessary details.  

- Reviewer expressed concerns about the limited evaluations and lack of zero-shot and CoT ablations. We clarified that we included GPT-4 evaluations, including zero-shot results. The paper already had an ablation study for LLaMA-3.1, strengthening our conclusions.

- Reviewer asked for clarification on the motivation for using specifications. We clarified that specifications mirror real-world optimization workflows, where problems are modified iteratively using high-level summaries instead of re-writing the entire problem (Sections 3.1 and 4.2).

- Reviewer noted the absence of an NL-to-MP baseline comparison. We clarified that our approach of modifying existing formulations aligns with practical workflows (which use specifications instead of NL problems), ensuring continuity and efficiency, unlike regenerating the MP for each modification, which would be inefficient and inconsistent.

**Outcome**: Reviewer didn't engage further and did not change their score.

**Chairs are encouraged to consult the detailed rebuttals for a deeper understanding of the points above**
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces WIQOR, a novel dataset for the task of Mathematical Program Modification (MPM), where optimization problems are revised based on natural language "what-if" questions. The task is novel, challenging, and aligns with real-world workflows, and experiments demonstrate its potential to advance NLP and operations research. Weaknesses include limited evaluation on a narrow set of models (mostly LLaMA variants, with GPT-4 added later), as well as concerns about the validity and diversity of generated problems. The presentation and writing are not well-prepared and complained by some reviewers. This paper receives overall negative scores of 6,3,5. Unfortunately, several reviewers did not participate actively in the discussion, and I have read the reviews and the authors’ comments. I think the authors have done a good job in the rebuttal, where they clarified some concerns on “novelty” and some unreasonable complaints on the metric. My major concern is on the limited evaluation present – as a benchmark paper, I think the current evaluations are not enough even though the authors added GPT-4 during rebuttal, more models such as mistral and qwen are required to be more comprehensive. Also, the writing and presentation of the paper has a large room to be improved. Therefore, I would like to recommend rejection of this paper.