You are an agent designed to solve constraint problems, you write and execute python code to answer questions. You have access to a python REPL, which you can use to execute python code. The python REPL will keep its state between usage. Use this to gradually approach a final solution. Divide the solution into meaningful parts run them part after part verifying that each part runs correctly. If you get an error, debug your code and try again only the last part that failed. You only use the constraint solver psychoco. Only use the output of your code to answer the question. Write for each code line a comment as needed to justify your reasoning for that code line. I will give you an example of good python code, please, follow the code convention of the *good python code example*. This *good python code example* is only an example. This *good python code example* has nothing to do with the constraint problem solution. You might know the answer without running any code, but you should still run the code to get the answer. If it does not seem like you can write code to answer the question, just return `"I don't know"` as the answer. The user may ask questions or give follow up instructions after you presented your solution. You then have to adjust your solution accordingly.

**good python code example:**

```python
"# Import necessary modules
import os

# Define the input and output file paths
input_file_path = \'numbers.txt\'
output_file_path = \'average.txt\'

# Function to read numbers from a file
def read_numbers_from_file(file_path):
    # Initialize an empty list to store the numbers
    numbers = []
    # Open the file in read mode
    with open(file_path, \'r\') as file:
        # Read each line in the file
        for line in file:
            # Strip any leading/trailing whitespace and convert to float
            number = float(line.strip())
            # Append the number to the list
            numbers.append(number)
    # Return the list of numbers
    return numbers
main()
```

<caption>**Figure 2:** Refining the prompt to provide comments for the generated model (slightly differently prompted).</caption>

positions in the prompt, as well as providing such an example in the prompt.

## 4.2. Prompting Specific Problems

The query part simply consists of the name of the constraint problem and related specific problem descriptions, see Listing 1¹².

### N-Queens Problem

**Problem:** Positioning n queens with possible movements known from chess on a *n*n chessboard<fnref n="13" />. Figure 1 shows the used prompt.

**Results:** The LLM makes multiple Python REPL calls, one that prints the solution, and one that follows the system prompt on how to verify the solution. This leads to the correct computation of the model including the conversion of the solutions to a list of lists (see Figure 3).

**Comments:** The solution to the N-queens problem is provided in the PyChoco documentation on the internet, hence, the LLM was trained with it. However, a different solution was found. Also, the 3-queen problem which provides no solution was correctly represented. A further observation was that the result is not deterministic, e.g., it varies arithmetical constraints for diagonals (see Listing 2). Fixing the seed and setting the temperature to zero does not make results deterministic.

The semantic verification test for the N-Queens problem is shown in Listing 3. Through the method

<footnote n="12">¹²The code in the repository contains further problems such as magic square that lead to similar observations.</footnote>

<footnote n="13">¹³See files `chats/queen8_cs_agent sol2 gpt-4-1106-preview_*` in the repository for results</footnote>
<caption>**Listing 1:** Queries for specific constraint problems</caption>

```python
queen3 = "solve the 3-queen problem"
queen8 = "solve the 8-queen problem"
queenn = "solve the n-queen problem, use n=8 as test instance"

coloring = """
solve the map coloring problem for four regions, three colors and the given adjacency:
regions = ['A', 'B', 'C', 'D']
adjacency_list = {
    'A': ['B', 'C'],
    'B': ['A', 'C', 'D'],
    'C': ['A', 'B', 'D'],
    'D': ['B', 'C']
}.

the solution should be a list of python dicts where each dict maps regions to color indices
"""
tsp = """solve the traveling salesman problem, use the following problem instance:
# Number of cities
C = 4

# Distance matrix
D = [[0, 10, 15, 20], [10, 0, 35, 25], [15, 35, 0, 30], [20, 25, 30, 0]]
the solution should be a list of valid solutions, each solution being a list of integers
representing the cities to be visited.
"""
```

**verify_solutions** called by the prompt, each solution (queens) is tested. This test is exhaustive because it analytically tests the queens' position, not leaving out a constraint.

### Map Coloring

**Problem:** Listing 1 describes the well-known Map Coloring as a query.

**Results:** A first model which contains the constraint `all不一样` raised an error<fnref n="14" /> but was corrected by using another constraint modeling (i.e., with `!=` instead of `all不一样`) through 2 iterations<fnref n="15" />. A further run provides a one-shot success, however not recognized as such, instead a solution was hallucinated for presenting to the user<fnref n="16" />. A further run firstly used the API wrongly, but finds finally a correct solution<fnref n="17" />.

**Comments:** The solution can also be found on the internet, however, this does not prevent from making errors. Finally, also because the iterative approach is used, a solution was found.

### Traveling Salesperson Problem

**Problem:** Listing 1 describes the well-known TSP as a query.

**Results:** In general, there are several recurring issues when attempting to model constraints for the TSP (see the error logs in Appendix Section 8). One major problem is accurately modeling distance constraints, which often leads to incorrect solutions. Similarly, the Hamiltonian path constraints are difficult to model correctly, compounding the problem. Additionally, there is frequent misuse of the API. These misuses typically involve providing arguments in the wrong order or supplying an incorrect number of arguments. As a result of these issues, the solutions generated are often flawed. With GPT-4, no solutions could be found with the resources at hand, but with GPT-4o, some solutions were correctly identified. However, no implementation was found that computes all the test cases provided for verification. Overall, these challenges underscore the complexity of effectively modeling TSP constraints and highlight areas that require significant improvements to find optimal solutions consistently.

## 4.3. Implementation

The implementation, which utilizes LangChain and Python, leverages ChatGPT-4 and ChatGPT-4o and is available on a repository at Zenodo<fnref n="18" />. As a constraint

<footnote n="14">¹⁴It seems to be an implementation issue, see https://github.com/chocoteam/choco-solver/issues/705</footnote>

<footnote n="15">¹⁵coloring_cs_agent sol2 gpt-4-1106-review _20240529094847.txt</footnote>

<footnote n="16">¹⁶coloring_cs_agent sol2 gpt-4-1106-review _20240529095157.txt</footnote>

<footnote n="17">¹⁷coloring_cs_agent sol2 gpt-4-1106-review _20240529095612.txt</footnote>

<footnote n="18">¹⁸https://doi.org/10.5281/zenodo.12794211</footnote>
Assistant:
There was a mistake in the code. It seems I incorrectly used the
'solver.findSolution)' method, which only finds one solution, and I
didn't convert the solution to the expected list format before passing
it to the verification function. I need to find all solutions and convert
each solution to a list of integers representing the positions of the
queens.
Let me correct the code.

Python_REPL (tool call 1/1):

from pychoco.model import Model
from test_tool import verify_solutions
model = Model("8-Queens")
# Create an array of 8 variables representing the positions of queens on a
# chess board
queens = model.intvars(8, 1, 8)
# Add constraints
for i in range(8):
    for j in range(i + 1, 8):
        # Queens cannot be on the same row
        model.arithmetic(queens[i], "!=", queens[j]).post()
        # Queens cannot be on the same diagonal
        model.arithmetic(queens[i], "-", queens[j], "!=", i - j).post()
        model.arithmetic(queens[i], "-", queens[j], "!=", j - i).post()
# Find all solutions
solver = model.get_solver()
solutions = []
for solution in solver.find_all_solutions():
    solutions.append([solution.get_int_val(queens[i]) for i in range(8)])
verify_solutions(solutions, "queen8")

Python_REPL (tool response):

92 of the 92 solutions are correct. Expected 92 solutions

Assistant:

The 8-queens problem has been successfully solved, and all 92 solutions are correct.

tool calls: 6, text messages: 6

<caption>Figure 3: Constraint model for 8-queens problem and last output</caption>

solver, we used PyChoco as a Python API for the con-
straint solver Choco<fnref n="19" />. The resources of ChatGPT are
limited in respect of context/prompt length. This ap-
proach causes the LLM to stop responding when the chat
size extends the allowed number of tokens, which easily
can occur because the chat includes the entire previous
chat history. The resulting chats are included in the pa-
per's appendix and the repository. The implementation
can be used to rerun the experiments, and the data can be
utilized for further investigation in prompt engineering.

5. Discussion

Overall, across all problems, after initiating multiple
chats, each with several iterations, a model that can com-

pute a single solution could be found (see Appendix sec-
tions 3 to 6). In Appendix Section 9, we document the
number of iterations per experiment, i.e., calls to the
Python interpreter ("Tool Calls") and answers given by
the LLM ("Text Messages"). Nevertheless, for the TSP, we
could not find a model that computes all solutions. The
iteration was stopped by the LLM, if no changes occurred
or if too many tokens were used. So the paper's result
is that in principle constraint models for well-known
problems can be computed, however, not in any cases.

An interesting observation is that the generated vari-
able names within the model are always semantically
meaningful (e.g., "queens") and not arbitrary. This likely
occurs because LLMs statistically favor names that have
been previously encountered.

In this paper, we use well-known constraint problems,

<footnote n="19">¹⁹https://choco-solver.org/</footnote>
<caption>Listing 2: Non-deterministic results</caption>

- correct
  model.arithmetic(queens[i], "!=", queens[j] + (j - i)).post()
  model.arithmetic(queens[i], "!=", queens[j] - (j - i)).post()
- correct
  model.arithmetic(qs[i], "-", qs[j], "!=", j - i).post()
  model.arithmetic(qs[j], "-", qs[i], "!=", j - i).post()
- wrong
  model.abs(queens - queens).ne(i - j).post() # Diagonal constraint
  model abs(queens - queens).ne(j - i).post() # Diagonal constraint
- wrong
  model.arithmetic(queens[i], "-", queens[j], "!=", i - j).post()
  model.arithmetic(queens[i], "+", queens[j], "!=", i + j + 2).post()
- does not find all solutions
  diag1 = [model.int_offset_view(queens[i], i) for i in range(n)]
  diag2 = [model.int_offset_view(queens[i], -i) for i in range(n)]
  model.all(diag1).post()
  model.all different(diag2).post()
- wrong
  model.arithmetic(queens[i], "!=", queens[j]).post() # Different columns
  model.arithmetic(queens[i], "-", i, "!=", queens[j], "-", j).post() # Different diagonals (left-top to right-bottom)
  model.arithmetic(queens[i], "+", i, "!=", queens[j], "+", j).post() # Different diagonals (left-bottom to right-top)
- wrong
  model.all different([queens[i], queens[j]]).post()
  model.arithmetic(queens[i], "-", queens[j], "!=", j - i).post()
  model.arithmetic(queens[i], "+", queens[j], "!=", j - i).post()

<caption>Listing 3: Verifying a solutions for N-queens through testing the queens' positions.</caption>

```python
def is_validSolution(queens):
    n = len(queens)
    assert type(queens) == list, "expected list of ints"
    for i in range(n):
        for j in range(i + 1, n):
            # Check if queens are in the same row
            if queens[i] == queens[j]:
                return False
            # Check diagonals: if the difference between the column indices equals
            # the difference between the row indices, they're in the same diagonal
            if abs(queens[i] - queens[j]) == abs(i - j):
                return False
    return True
```

which, of course, can only be a first step for leveraging LLMs in knowledge acquisition. The next steps would be to supply new, unknown problems, also industrial ones, e.g., by giving tables of correct variable combinations (configurations) in table constraints and generating abstracted constraint models. However, Listing 1 shows a potential way of representing formally a problem as a query which probably can be used for other tasks.

There are several general problems identified: the program persistently uses the PyChoco API incorrectly and often hallucinates PyChoco functions. This could not be

prevented by providing the API definition in the prompt, because the complete definition extends the number of possible tokens we could send to the LLM.

Additionally, the chats are not stable and can vary significantly even with the same prompt. This instability persists even with temperature set to 0 and a fixed seed, making chats non-deterministic.

The used prompts should be further developed to achieve the intended general generation of knowledge models; in other words, prompt engineering for knowledge modeling needs to be elaborated. Few-Shot Prompt-
ing [20] could be used to encourage the model to produce more consistent outputs and adhere more closely to given instructions.

The approach of first creating syntactically correct models and then verifying them to ensure semantic accuracy could be successfully demonstrated (see Listing 3). Because we use a test-driven semantic verification this verification depends on the quality of the used test cases. For example, the N-queen problem was exhaustive because of an analytical-based verification and not an enumeration of test cases. Furthermore, writing tests for knowledge-based tasks is, in our experience, typically easier than writing knowledge models, hence, the experts' tasks of creating formal constraints and also configuration models will, in our opinion, shift from writing models to writing tests - and let us write the models through an LLM. Thus, the knowledge model developing process probably, with the presented approach, can be executed by domain experts not only by knowledge engineers.

The paper offers, furthermore, an easy way to begin with iterative LLM prompting. Moreover, it is suggested to use the LLM for creating a new configuration language by exploiting the hallucination capabilities.

## 6. Conclusions

The paper presents the first steps in supporting the formalization task in constraint modeling with LLMs, i.e., the mapping of a well-known problem into a constraint representation of a certain constraint language, here PyChoco, with large language models. By taking various constraint problems, formulating generally applicable prompts, and doing so in an iterative manner which includes syntactic and semantic verification processes, the LLM could generate appropriate and correct constraint models. Through the iteration, in-between errors related to syntax and semantics were automatically corrected. Further work will include a quantitative evaluation of the method, a comparison of different LLMs, as well as the generation of constraint models for unknown problems. In total, this paper provides a first step into a new kind of constraint and configuration modeling approach with LLMs.

## Acknowledgments

This project was funded through the Federal Institute for Research on Building, Urban Affairs and Spatial Development (Bundesinstitut für Bau-, Stadt- und Raumforschung), Germany, within the Zukunft Bau Forschungsförderung Maßnahme (Future of construction Research funding Measure): "AI-supported modernization of rental housing stock" ("KI-gestützte Modernisierung an Mietwohnungsbaubeständen (KIMM)") as well as through the Federal Ministry for Economic Affairs and Climate Action (Bundesministerium für Wirtschaft und Klimaschutz), Germany, within the "Joint project: EnOB: IntelMOD - Intelligent modernization platform based on functional cost splitting; sub-project: Infrastructure of the modernization platform." ("Verbundvorhaben: EnOB: IntelMOD - Intelligente Modernisierungsplattform auf Basis des Funktionalen Kostensplittings; Teilprojekt: Infrastruktur der Modernisierungsplattform.")

## References

[1] A. Felfernig, L. Hotz, C. Bagley, J. Tiihonen, Knowledge-based Configuration - From Research to Business Cases, Morgan Kaufmann, 2014.

[2] A. Felfernig, A. Falkner, D. Benavides, Feature Models: AI-driven Design, Analysis, and Applications, Springer, 2024.

[3] L. Hotz, A. Felfernig, M. Stumptner, A. Ryabokon, C. Bagley, K. Wolter, Configuration Knowledge Representation & Reasoning, in: A. Felfernig, L. Hotz, C. Bagley, J. Tiihonen (Eds.), Knowledge-based Configuration - From Research to Business Cases, Morgan Kaufmann Publishers, 2014, pp. 59-96.

[4] I. Ozkaya, A. Carleton, J. Robert, D. Schmidt, Application of large language models (llms) in software engineering: Overblown hype or disruptive change?, Carnegie Mellon University, Software Engineering Institute's Insights (blog), 2023. URL: https://doi.org/10.58012/6n1p-pw64, accessed: 2024-May-16.

[5] B. O'Sullivan, Automated modelling and solving in constraint programming, in: 24th AAAI Conference on Artificial Intelligence, AAAI'10, AAAI Press, 2010, pp. 1493-1497.

[6] E. C. Freuder, Conversational modeling for constraint satisfaction, AAAI Conference on Artificial Intelligence 38 (2024) 22592-22597. URL: https://ojs.aaai.org/index.php/AAAI/article/view/30268. doi:10.1609/aaai.v38i20.30268.

[7] F. Petroni, T. Rocktäschel, P. P. Lewis, A. Bakhtin, Y. Wu, A. Miller, S. Riedel, Language models as knowledge bases?, 2019. URL: https://arxiv.org/pdf/1909.01066.

[8] Y. Ding, A. Poudel, Q. Zeng, T. Weninger, B. Veeramani, S. Bhattacharya, Entgpt: Linking generative large language models with knowledge bases, 2024. URL: https://arxiv.org/abs/2402.06738.

[9] S. Razniewski, A. Yates, N. Kassner, G. Weikum, Language models as or for knowledge bases, 2021. URL: https://arxiv.org/abs/2110.04888.

[10] F. Suchanek, A. T. Luu, Knowledge bases and language models: Complementing forces, in: 7th Inter-
national Joint Conference on Rules and Reasoning,
Springer-Verlag, Berlin, Heidelberg, 2023, pp. 3–15.
URL: https://doi.org/10.1007/978-3-031-45072-3_1.

[11] A. Nayak, H. P. Timmapathini, Llm2kb: Constructing knowledge bases using instruction tuned context aware large language models, 2023. URL: https://arxiv.org/abs/2308.13207.

[12] T. Ahmed, S. Choudhury, Lm4opt: Unveiling the potential of large language models in formulating mathematical optimization problems, 2024. URL: https://arxiv.org/abs/2403.01342.

[13] Y. Zhang, S. Mao, T. Ge, X. Wang, A. de Wynter, LLM as a mastermind: A survey of strategic reasoning with large language models, 2023. URL: https://arxiv.org/abs/2404.01230v1. arXiv:2404.01230.

[14] W. Chen, X. Ma, X. Wang, W. W. Cohen, Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks, Transactions on Machine Learning Research (2023). URL: https://openreview.net/forum?id=YfZ4ZPt8zd.

[15] L. Pan, A. Albalak, X. Wang, W. Y. Wang, Logiclm: Empowering large language models with symbolic solvers for faithful logical reasoning, 2023. arXiv:2305.12295.

[16] P. Clark, O. Tafjord, K. Richardson, Transformers as soft reasoners over language, in: 29th International Joint Conference on Artificial Intelligence, IJCAI'20, 2021, pp. 3882–3890.

[17] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. H. Hsin Chi, D. Zhou, Self-consistency improves chain of thought reasoning in language models, ArXiv (2022). URL: https://arxiv.org/abs/2203.11171.

[18] M. Acher, J. G. Duarte, J.-M. Jézéquel, On programming variability with large language model-based assistant, in: Proceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A, SPLC '23, Association for Computing Machinery, New York, NY, USA, 2023, p. 8–14.
URL: https://doi.org/10.1145/3579027.3608972.

[19] M. Funk, S. Hosemann, J. C. Jung, C. Lutz, Towards ontology construction with language models, 2023. URL: https://arxiv.org/abs/2309.09898.

[20] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sasety, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems 33 (2020) 1877–1901.
# Configuration Copilot: Towards Integrating Large Language Models and Constraints

Philipp Kogler¹,*, Wei Chen¹, Andreas Falkner¹, Alois Haselböck¹ and Stefan Wallner¹

¹Siemens AG Österreich, Siemensstraße 90, 1210 Wien, Austria

**Abstract**

A product configurator enables the configuration of a customizable product while constraining possible variations. Users typically interact with a product configurator via a graphical user interface. A complex product can be composed of components and parameters that are not easily understandable for non-experts which can prevent them from effectively configuring the product. In this paper, we propose a configuration copilot, an interactive chat-based interface that allows users to iteratively configure a product by describing their requirements in natural language. Our framework leverages the Natural Language Processing (NLP) capabilities of advanced pre-trained Large Language Models (LLMs) alongside the robustness of constraint-based product configurators. We introduce a technical architecture that accurately formalizes constraints from natural language inputs, identifies valid product configurations based on a defined product line and specified constraints using a constraint solver, and communicates the resulting product configurations back to the end user in natural language. We demonstrate and evaluate the configuration copilot on two use-cases: The configuration of the GoPhone feature model (Boolean feature assignments), and the configuration of a metro wagon (more general configuration parameters).

**Keywords**

Product Configuration, Constraints, Feature Models, Large Language Models, Copilot

## 1. Introduction

Product configuration involves creating customized products from predefined components while satisfying constraints that limit configurable parameters and possible combinations [1]. A product configurator is a software tool that allows users to configure a product, commonly through a graphical user interface and often in a web-based context. Therefore, interface and interaction design plays a major role in the development of a product configurator but is often overlooked [2]. This observation is especially relevant when complex products are configured by non-expert users. The meaning of configurable components and parameters may not be obvious which prompts a need for explanation and introduces a learning curve.

As an alternative to GUI-based interactions with product configurators, we propose a configuration copilot that offers a text-based chat interface. Uninformed users shall be able to describe their requirements in natural language without knowledge of the concrete parameters to set and components to select. The copilot shall then configure the product and respond with a valid configuration

complying with the initial requirements. The user shall be able to interactively refine the product configuration.

We utilize a pre-trained Large Language Model (LLM) for the processing of natural language. Recent advances in this field have enabled use cases that require the understanding and generation of not only natural language but also code. Well-known limitations include a lack of reliability, guaranteed correctness, domain-specific knowledge in general-purpose LLMs, and limited reasoning abilities [3]. In our configuration copilot, we address these shortcomings by combining a LLM with a constraint solver. While the strengths of the LLM are utilized in the processing of the natural-language requirement descriptions, the reasoning to find valid configurations is done by the constraint solver.

In this paper, we first describe LLMs and constraint-based product configuration in Section 2 and related work in Section 3. We detail the technical architecture of the configuration copilot in Section 4, and present an evaluation based on the two use-cases of configuring the GoPhone feature model and a metro wagon in Section 5. We conclude the paper with a summary, a limitation statement, and future work in Section 6.

## 2. Background

### 2.1. Large Language Models

Pre-training task-agnostic aspects of natural language processing (NLP) tasks is a central concept of LLMs. The Transformer architecture enables this approach on a large scale through parallelization. Transformer

ConfWS'24: 26th International Workshop on Configuration, Sep 2-3,
2024, Girona, Spain

*Corresponding author.

<figure><img src="image_50.png" /></figure>

philipp.kogler@siemens.com (P. Kogler);

chen.wei@siemens.com (W. Chen); andreas.a.falkner@siemens.com (A. Falkner); alois.haselboeck@siemens.com (A. Haselböck);

stefan.wallner@siemens.com (S. Wallner)

<figure><img src="image_51.png" /></figure>

0009-0009-5598-1225 (P. Kogler); 0009-0008-0486-9068

(W. Chen); 0000-0002-2894-3284 (A. Falkner); 0000-0003-2599-3902 (A. Haselböck); 0000-0002-9755-6632 (S. Wallner)

<figure><img src="image_52.png" /></figure>

© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License
Attribution 4.0 International (CC BY 4.0).
models are able to capture complex patterns and long-range-dependencies in texts through the multi-head self-attention mechanism. Compared to previous state-of-the-art models such as recurrent neural networks (RNNs) or long short-term memory networks (LSTMs) a performance improvement in various NLP tasks is observed [4, 5].

Decoder-only models are a subclass of Transformer-based architectures and are primarily used for sequence-to-sequence tasks such as translation. Auto-regressive models predict the next single token (sub-word) by maximizing the log-likelihood given all previous words and the model parameters [4].

The size and quality of the pre-training corpus have a strong impact on performance [5]. LLMs are trained on publicly available data and excel in general language tasks. Highly specialized tasks require expert knowledge that is often not included in the training data, and therefore LLMs may not be able to generate accurate output. Task-specific knowledge can be introduced to a general-purpose LLM through domain customization by employing techniques like prompting and fine-tuning [6].

## 2.2. Constraint-based Product Configuration

Product configuration involves selecting and assembling various components and options to meet customer requirements and constraints. Its complexity arises from the vast number of possible combinations and the need to satisfy all technical restrictions and customer preferences. To handle this complexity, powerful technologies have been developed and established in the last decades. Constraint-based systems shall be highlighted here, which allow to represent the product line and its technical restrictions and requirements in a clean, logical way, thereby ensuring that only valid configurations are generated. The core of such systems lies in the ability to handle complex and combinatorial search spaces efficiently through the use of advanced solving algorithms, such as backtracking, forward checking, and constraint propagation. This facilitates the efficient generation of feasible solutions while pruning invalid combinations.

An important subdomain of configuration problems are feature models for the representation of product lines [7]. Constraint-based techniques are especially well-suited for such feature models, because of the simple language and the mainly Boolean type of the variables.

MiniZinc is a constraint language that can be used to represent configuration problems [8]. Several efficient solvers can process this language and can therefore be used as the backend of a configurator.

A product configurator is almost always an interactive system [9]. A graphical user-interface (GUI) allows to enter the user requirements, which are passed on as input

to the constraint solver. The results of the solver are presented on the GUI, and the user can vary or refine her/his input specification and the solver is called again.

To design and implement a configurator GUI can be a challenging task, because the possible interactions are diverse, like collecting the requirements, reporting invalid constellations, representing a solution, showing a performance value of a solution, etc. In addition, every modification of the product (line) requires a review and possibly and adjustment of the GUI.

In the following sections, we demonstrate how to eliminate the need for a product-specific GUI by utilizing an LLM to engage in dialogue with the user.

# 3. Related Work

Various approaches to improve the reliability, the performance in domain-specific tasks, and the reasoning abilities of LLMs are described in literature.

Few-shot prompting effectively introduces domain-specific knowledge and improves the task-specific performance of LLMs by adding a small set of example interactions (input and expected output) to the prompt [10]. Chain-of-thought prompting was shown to improve the reasoning abilities of LLMs especially in more complex tasks by providing exemplary intermediate reasoning steps [11].

Grammar prompting is used when a specific output format is expected. Wang et al. describe how a minimal specialized grammar is obtained in a grammar specialization process by selecting a specialized grammar as a subset of the full grammar using an LLM and minimizing it by parsing the output and forming the union of used rules. In their approach, constrained decoding then validates the output syntax [12]. Similarly, Poesia et al. presented the Synchromesh framework: Using a few-shot prompting technique, semantically similar examples are selected from a larger pool for a given natural language prompt via a similarity metric named Target Similarity Tuning. Constraints are enforced through Constrained Semantic Decoding to verify syntax validity, scoping, or type checks. During the token-by-token construction of the LLM output, a Completion Engine provides all valid tokens that can further extend a partial program towards a full correct program [13].

Neuro-symbolic approaches focus on combining the strengths of neural networks and symbolic reasoners. Pan et al. introduced the Logic-LM framework which achieves a performance improvement of 18% on logical reasoning datasets over chain-of-though prompting. The framework translates the natural-language input into symbolic formulations and utilizes a symbolic reasoner to obtain the answer [14].

This paper builds upon our previous work [15] that
studied the reliable generation of formal specifications with LLMs using algorithmic post-processing. We extend the approach towards product configuration by applying post-processing to reliably integrate a constraint solver. In addition to previously described guaranteed syntactically valid output, this extension enables arbitrary semantic constraints.

# 4. Configuration Copilot

This section presents the technical details of the configuration copilot that combines LLMs with constraint-based configuration.

## 4.1. Architecture

Figure 1 shows an overview of the architecture. A user configures a product by providing a natural-language description of their requirements. The Formalizer (see Section 4.2) is a specialized LLM-based component that translates the requirements to constraints. The Configuration Engine is a constraint solver that attempts to find a configuration that satisfies the general constraints of the product line combined with the user constraints provided by the Formalizer. An Interpreter (see Section 4.4) translates the configuration back to natural language. The configuration copilot then responds with a natural-language description of the configured product accompanied by the full technical specification (product configuration as determined by the Configuration Engine). The user can then further refine the product configuration interactively.

<figure><img src="image_53.png" /><figcaption>Figure 1: Architecture of the Configuration Copilot</figcaption></figure>

## 4.2. Formalizer

The input to the Formalizer is a natural-language description of arbitrary product requirements provided by a non-expert. Utilizing the NLP capabilities of LLMs, the formalization can be viewed as a sequence-to-sequence translation task from natural language to a formal specification. The LLM is tasked with natural language understanding and the identification of corresponding parameters or components of the product (line), but is specifically not tasked with reasoning (e.g., constraint satisfaction). While pre-trained LLMs achieve a strong performance on general tasks, they do not have knowledge of the specific product (line) to configure as corresponding data is not included in their training corpus [5, 6]. Additionally, the probabilistic nature of the token-by-token output construction of LLMs does not provide any guarantees in the correct generation of valid constraints [4].

Our framework for reliable code generation addresses domain-customization and reliable output generation through few-shot prompting and algorithmic post-processing [15].

Few-shot prompting has been shown to effectively extend the capabilities of LLMs with domain knowledge while requiring significantly less training data than fine-tuning [10]. Knowledge of the product line is incorporated through a system prompt describing the product line with its parameters and components. A small set of examples is appended as pairs of natural-language inputs and expected outputs to provide the LLM with more context and guide it towards the expected behavior.

Rather than generating output directly in a specific constraint language, an intermediary JSON-based language is used, which can then be easily transpiled. The transpile parses the JSON constraint representation and maps its elements to corresponding constructs of the specific constraint language following predefined rules. As JSON is widely used, pre-trained LLMs have more often encountered JSON than less common constraint
languages. Therefore, the generation of an intermediary
JSON output is closer to the LLMs capabilities. Addition-
ally, an intermediary language gives more control over
the expected output as available language constructs can
be constrained and tailored to the specific task. It also
decouples the Formalizer from the Configuration Engine
by enabling interchangeability of the concrete constraint
language. To generate a valid JSON for the Formalizer,
several state-of-the-art LLMs are evaluated and bench-
marked. Specialized code LLMs that are pre-trained on
the translation of natural language to code in a variety of
programming languages are believed to be more suitable
for the generation of structured JSON output. In our eval-
uation in Section 5, we selected four open-access LLMs:
Two code LLMs (CodeLLama [16] and Codestral [17]),
and two general-purpose instruction-tuned LLMs (Meta
Llama 3 [18] and Mistral [19]).

Algorithmic post-processing guarantees the correct generation of the JSON-based intermediary language and is depicted in Figure 2. As the auto-regressive Transformer model generates its output step-by-step as tokens, the post-processor engages into every generation step: For each step, the LLM generates a list of candidates for the next token based on the prompt and the generated output so far. Sorted by priority as evaluated by the LLM, the post-processor determines whether the token candidate represents a valid continuation of the partial output sequence (partial intermediary JSON). The valid token candidate with the highest priority is then selected, handed back to the LLM, and added to the partial JSON, extending it one step further. A completeness checker evaluates after every step to determine, if the JSON is complete [15].

The JSON-based intermediary language is formally
defined by a JSON schema specification and the post-
processor is therefore a specialized JSON validator that
can strictly validate any partial JSON against the schema.
This implementation is based on deterministic finite au-
tomata (DFA). Each generic JSON language element (ob-
ject, list, string, number, etc.) is represented by a DFA,
keeping track of the current state. The token generated
by the LLM is broken down to single-character inputs
for the JSON validator. Depending on the schema and
the current state, only a set of characters is accepted. If
a character is rejected, the current token is considered
invalid, and the validator state is rolled back to the last
valid token. State changes are triggered by characters
until the final state is reached. When the DFA reaches
its final state, the generated valid JSON is complete [15].

4.3. Configuration Engine

Given the user constraints combined with the complete
product line definition, the Configuration Engine evalu-
ates whether the constraints are satisfiable and returns

a configuration. The product line as well as the user
constraints are modelled in the MiniZinc constraint lan-
guage [8]. The solver returns the full product configura-
tion as a list of variable assignments which serves as an
input to the Interpreter. In this context, we consider the
constraint solver a given technology that will neither be
further described nor evaluated.

4.4. Interpreter

The Interpreter is an LLM module that explains the prod-
uct configuration found by the Configuration Engine.
The goal is to provide the user with a less technical sum-
mary that is understandable for non-experts.

Structured few-shot prompting [10] is sufficient for
this use-case as LLMs generally perform well in the trans-
lation from a formal specification to a natural-language
summary as all facts are directly present in the prompt.
The context given to the LLM consists of three aspects:
The product line definition, instructions, and examples.
The LLM is prompted to evaluate which properties and
components are most important to be included in the
summary. This is achieved by adding importance hints
to the product line definition, and by appending the orig-
inal user input. Properties and components mentioned
directly in the user input are given more importance and
are more likely to be included in the summary. The result
is a more natural context-aware explanation of the most
relevant aspects in the product configuration.

5. Evaluation

The presented configuration copilot is evaluated on two
use-cases: The conceptually simpler task of configuring
a feature model, and the configuration of a metro Wagon.

5.1. Feature Model (GoPhone)

The first use-case for the evaluation of the presented copilot is the configuration of a feature model. An uninformed user shall be supported in the configuration of the GoPhone from the SPLOT project [20].

The GoPhone is a feature model comprised of 77 fea-
tures with some being mandatory, optional, dependent
on other features, or mutually exclusive. For example,
the feature *call* is mandatory for the GoPhone, the fea-
ture *accept_incoming_call* is mandatory for *call*, but
*show_missed_calls* and *show_received_calls* are op-
tional.

Feature assignments are Boolean, either the feature is
included in the product configuration (true) or the fea-
ture is not included (false). The product line definition
is a MiniZinc program that was directly derived from the
<figure><img src="image_54.png" /><figcaption>Figure 2: Detail view of the Formalizer with post-processing</figcaption></figure>

feature model. Each feature is a Boolean variable. Constraints limit the combination of features and therefore limit possible product configurations.

A non-expert user starts by describing their requirements for the phone in natural language:

> I need a basic phone to call people and browse the web but I don't play games. I also want to keep track of my appointments.

This natural-language description is then formalized to the intermediary JSON language:

```json
{
  "features": [
    {
      "name": "make_call",
      "value": true
    },
    {
      "name": "browsing",
      "value": true
    },
    {
      "name": "game",
      "value": false
    },
    {
      "name": "calendar_entry",
      "value": true
    }
  ]
}
```

This list of solver-independent constraints is then transpiled to MiniZinc constraints:

```c
constraint make_call = true;
constraint browsing = true;
constraint game = false;
constraint calendar_entry = true;
```

Together with the MiniZinc program (product line definition), the Configuration Engine evaluates the constraints and returns a full product configuration of the GoPhone for the specific user requirements as a list of Boolean feature assignments. In this work, the Gecode [21] solver was used without further configuration or optimization. The Interpreter converts this configuration back to natural language and returns it to the user. An example for such an output is (the technical specification is shortened for brevity):

> Your GoPhone can manage ringing tones, messages, and browse the web. It can also manage calls, read multimedia, and display photos. It has a calendar entry feature and an address book processing system. However, it does not play games, organize tasks, or have currency conversion features.

> Here is the full technical configuration:

```
GoPhone = true;
manage_ringing_tones = true;
[...]
browse = true;
[...]
game = false;
play games = false;
install games = false;
[...]
```

The crucial and potentially failing component of the presented architecture is the Formalizer: the probabilistic nature of the underlying LLMs does not provide strict guarantees. Especially the translation of the user's requirements to the feature assignments is subject to uncertainty. A formal evaluation of the Interpreter is not done because the correctness requirements for the configuration summary are less strong and LLMs are generally known to perform well on simple summarization tasks when the facts are directly provided. It is also unsuitable
to define a single reference solution as a large variety of summaries (with various feature assignments being explained or not explained) could be considered correct. Ultimately, users need to decide whether the summary was helpful or not.

The Formalizer was evaluated on a custom dataset of 30 test cases. 15 test cases create a new configuration from scratch, and 15 test cases evaluate a re-configuration where a given configuration is modified. Each test case consists of natural-language input mentioning between two and six feature requirements in the text (and up to 30 given feature assignments for modification test cases), and the expected feature assignments in JSON. Using the natural-language input, the Formalizer generates feature assignments in JSON. This output is compared to the expected output. The comparison is conceptually challenging due to the intrinsic ambiguity of natural language. In many cases, one could argue for multiple options of feature assignments to be considered a correct translation. In this evaluation, we hand-crafted the dataset to be less ambiguous. However, the features of the GoPhone are in themselves sometimes not obviously distinguishable, and multiple features may be equally suitable. For example, the feature browsing is an optional sub-feature of the more general parent feature browse. This ambiguity was addressed by encoding very similar features to the same representation. Therefore, all defined synonymous features are considered a correct feature assignment for a requirement. However, the feature assignment was not limited to leaf features because doing so would add reasoning requirements to the Formalizer. Consider the leaf features play games and install games, and the parent feature game. If a user only mentions games in their descriptions, the more abstract feature game shall be assigned. Otherwise, the LLM would have to reason about a proper assignment of leaf features, deviating from the most direct translation from natural language to a feature assignment. The reasoning regarding further (sub-)feature assignments shall be done by the Configuration Engine.

A similarity metric based on the Jaccard distance between sets [22] was used to compare each pair of expected and actual output: Let $T$ and $F$ be the sets of feature names in the expected output where the feature value is True and False, respectively. Similarly, let $\hat{T}$ and $\hat{F}$ be the sets of feature names in the actual output where the feature value is True and False, respectively. The Jaccard similarities are:

For the True sets:

$$S_T = \frac{|T \cap \hat{T}|}{|T \cup \hat{T}|}$$

For the False sets:

$$S_F = \frac{|F \cap \hat{F}|}{|F \cup \hat{F}|}$$

<caption>**Table 1**
Evaluation Results for the GoPhone Formalization</caption>

S = Similarity score

F1 = F1 score

<table><thead><tr><th>Model [Size/Quantization]</th><th>S</th><th>F1</th></tr></thead><tbody><tr><td>CodeLlama 34B/Q4 [Link]</td><td>0.65</td><td>0.74</td></tr><tr><td>Codestral 22B/Q4 [Link]</td><td><strong>0.79</strong></td><td><strong>0.86</strong></td></tr><tr><td>Meta Llama 3 8B/Q8 [Link]</td><td>0.46</td><td>0.58</td></tr><tr><td>Mistral 7B/Q8 [Link]</td><td>0.69</td><td>0.79</td></tr></tbody></table>

The overall similarity between the expected and actual output S as the weighted average of $S_T$ and $S_F$ is:

$$S = \frac{S_T \cdot |T \cup \hat{T}| + S_F \cdot |F \cup \hat{F}|}{|T \cup \hat{T}| + |F \cup \hat{F}|}$$

The result is a number between 0 and 1, with 0 indicating no similarity, and 1 indicating a perfect match. In this metric, the identification of features in the natural language as well as the Boolean assignment are considered.

Similarly, the precision $P$, recall $R$ and F1 score $F1$ were calculated:

$$P = \frac{|T \cap \hat{T}| + |F \cap \hat{F}|}{|\hat{T}| + |\hat{F}|}$$

$$R = \frac{|T \cap \hat{T}| + |F \cap \hat{F}|}{|T| + |F|}$$

$$F1 = 2 \cdot \frac{P \cdot R}{P + R}$$

We selected four open-access LLMs from HuggingFace to be evaluated in the context of the configuration copilot, two code models, and two general-purpose models. Table 1 summarizes the evaluation results for the GoPhone use-case per LLM. Codestral 22B/Q4, a state-of-the-art code model, performed best. However, Mistral 7B/Q8 outperformed the larger code model CodeLlama 34B/Q4 against our expectations. This shows that the performance of LLMs is use-case specific and must be evaluated. We found that the performance degrades as instances become more complex. Remedies for this observation are the use of larger models, tuning the technical approach, or future improvements of LLMs themselves. Considering the remaining ambiguity of natural language, the results indicate reasonable performance in this use-case as the majority of feature requirements was formalized correctly.

## 5.2. Metro Wagon

The second use-case for the evaluation is a metro Wagon configuration problem (see [23]) that uses not only Boolean but also numeric variables and arrays, where a configurable product has components that can occur
<figure><img src="image_55.png" /><figcaption>**Figure 3:** Class diagram of the Wagon example. Default values are underlined. Wagon.all-equal-type() stands for a constraint that all sub-parts must have the same type except for special. Wagon.all-equal-color() stands for a constraint that all associated seats (except if type=special) must have the same color.</figcaption></figure>

```json
{
  "nr_passengers": {
    "type": "greaterThan",
    "value": 120
  },
  "nr_seats": {
    "type": "equals",
    "value": 40
  },
  "seat_color": [
    "red",
    "red",
    "red",
    ...
  ]
}
```

This list of solver-independent constraints is then transpiled to MiniZinc constraints:

```c
constraint nr_passengers > 120;
constraint nr_seats = 40;
constraintforall (i in 1..nr_seats)
    (seat_color[i] = red);
```

multiple times (similar to generative constraint satisfaction [24] or cardinality-based feature modelling [25]).

A metro train wagon has as configurable attributes the size (length in millimetres: 10000..20000) and the expected load (number of passengers: 50..200) which can be realized as seats or standing room. As components we consider only seats (max. 4 per meter of length) and handrails, and their number is configurable.

There is at most one handrail in a wagon (mandatory if there is standing room) and it has a configurable type: "standard" or "premium".

A single seat consumes standing room for 3 persons and has as configurable attributes the type ("standard", "premium", "special") and the color ("blue", "red", "white"). The type is constrained such that standard is not allowed to be mixed with premium (for seats and handrails). The color of all seats must be the same, except for special seats which have to be "red".

Figure 3 shows a UML class diagram for this sample specification, including pseudo code for all constraints.

A non-expert user starts by describing their requirements for the metro Wagon in natural language:

> The wagon should accommodate more than 120 people with room for 40 to sit. Seats should be red.

This natural-language description is then formalized to the intermediary JSON language:

Together with the MiniZinc program (product line definition), the Configuration Engine evaluates the constraints and returns a full product configuration of the metro Wagon for the specific user requirements as a list of value assignments to the configurable parameters. The Interpreter converts this configuration back to natural language and returns it to the user:

> Your metro Wagon is 20 meters long, has space for 160 passengers with 40 red standard seats and a standard handrail. There is also standing room for an additional 120 people.

> Here is the full technical configuration:

```c
length_mm = 20000;
nr_passengers = 160;
nr_seats = 40;
standingulu = 120;
nr_handrails = 1;
handrail_type = standard;
seat_color = [red, red, red, ...];
seat_type = [standard, standard, standard, ...];
```

The Formalizer for the metro Wagon use-case was evaluated, like the GoPhone Formalizer, on a diverse set of 30 test cases (pairs of input and expected output) with 15 creating a new configuration and 15 modifying a given configuration (re-configuration). To evaluate the similarity in this use-case, the previously described similarity metric based on the Jaccard distance between sets for Boolean feature assignments was extended to the more general use-case. This extension is necessary to enable the evaluation of the value assignment for the
<table><caption>**Table 2**

Evaluation Results for the Metro Wagon Formalization
S = Similarity score</caption><thead><tr><td>Model [Size/Quantization]</td><td>S</td></tr></thead><tbody><tr><td>CodeLlama 34B/Q4 [Link]</td><td>0.77</td></tr><tr><td>Codestral 22B/Q4 [Link]</td><td><b>0.78</b></td></tr><tr><td>Meta Llama 3 8B/Q8 [Link]</td><td>0.68</td></tr><tr><td>Mistral 7B/Q8 [Link]</td><td>0.72</td></tr></tbody></table>

extended variable types (i.e., strings, numbers, arrays). While the Jaccard distance remained the basis for the similarity metric, a type-specific value metric was applied to each configuration parameter that is present in both, the expected and the actual output. In addition to the parameters being present, the total similarity is adjusted according to the value similarities as well. The type-specific metric considers:

* for numeric values: operator ('=', '>', '<', etc.) and value distance relative to the parameter-specific domain (value range)

* for array values: length and positional item equality

* for string-enumerated values: exact value match

Let C be the set of configuration parameter names in the expected output and let $\hat{C}$ be the set of configuration parameter names in the actual output. The Jaccard similarity $S_J$ is:

$$S_J = \frac{|C \cap \hat{C}|}{|C \cup \hat{C}|}$$

Let c be a matching parameter that is in both, the expected and the actual output, and let $S_v(c)$ be the type-specific value similarity (between 0 and 1) of c between the expected and the actual output. The value-adjusted Jaccard similarity S is then:

$$S = \frac{\sum_{c \in C \cap \hat{C}} S_v(c)}{|C \cup \hat{C}|}$$

The evaluation of the F1 score is omitted because it does not provide any additional value, as it appears to correlate strongly with the already rather strict similarity score S. Table 2 summarizes the evaluation results for the metro Wagon use-case per LLM. Codestral 22B/Q4 performed best again with a similar score. However, the other three models consistently improved their score compared to the GoPhone use-case. While the metro use-case in itself is more complex, the domain size (amount of named parameters) is lower, which may be the reason for the higher performance. Overall, the results again indicate a reasonable performance for the metro Wagon use-case.

# 6. Conclusion

This paper presented a configuration copilot that enables non-expert users to configure a product in natural language. The cooperative neuro-symbolic approach combines an LLM with a constraint solver to reliably support a product configuration. An early evaluation on the two use-cases of configuring the GoPhone feature model and a metro Wagon indicated practical feasibility. We believe that a configuration copilot is a valuable extension to GUI-based product configurators. For a productive implementation, limitations and future work mentioned in Sections 6.1 and 6.2 should be addressed.

## 6.1. Limitations

A limitation of our work is the size of the use-cases. Compared to real-world scenarios, the evaluated GoPhone feature model and metro Wagon are smaller and less complex. Additionally, the evaluation was done on a limited manually created dataset with 30 instances per use-case. While the most critical aspect of the architecture, the Formalizer, was evaluated, a formal evaluation of the Interpreter and the full configuration pipeline were omitted for the reason that a user study is required to evaluate these aspects. This paper demonstrates that creating a productive configuration copilot is feasible but does not study the extent to which value is provided to real users in a real-world scenario.

## 6.2. Future Work

To address the limitations of this paper, the configuration copilot shall be evaluated on more complex use-case from practice in a user study. The configuration copilot itself shall be extended: When a configuration as specified by the user is unsatisfiable, the configuration copilot shall suggest alternatives instead of reverting to the last satisfiable configuration. Additionally, soft constraints in the form of 'If possible, I would like to ...' shall be introduced.

# References

[1] L. Zhang, Product configuration: A review of the state-of-the-art and future research, International Journal of Production Research 52 (2014) 6381-6398. doi:10.1080/00207543.2014.942012.

[2] M. Yi, Z. Huang, Y. Yu, Creating a sustainable e-commerce environment: The impact of product configurator interaction design on consumer personalized customization experience, Sustainability 14 (2022). URL: https://www.mdpi.com/2071-1050/14/23/15903. doi:10.3390/su142315903.
[3] E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, C. Xiong, Codegen: An open large language model for code with multi-turn program synthesis, in: The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, OpenReview.net, 2023.

[4] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, I. Polosukhin, Attention is all you need, in: I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, R. Garnett (Eds.), Advances in Neural Information Processing Systems, volume 30, Curran Associates, Inc., 2017. URL: https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.

[5] B. Min, H. Ross, E. Sulem, A. P. B. Veyseh, T. H. Nguyen, O. Sainz, E. Agirre, I. Heintz, D. Roth, Recent advances in natural language processing via large pre-trained language models: A survey, ACM Computing Surveys (2023).

[6] C. Ling, X. Zhao, J. Lu, C. Deng, C. Zheng, J. Wang, T. Chowdhury, Y. Li, H. Cui, X. Zhang, T. Zhao, A. Panalkar, W. Cheng, H. Wang, Y. Liu, Z. Chen, H. Chen, C. White, Q. Gu, C. Yang, L. Zhao, Beyond one-model-fits-all: A survey of domain specialization for large language models, CoRR abs/2305.18703 (2023). URL: https://doi.org/10.48550/arXiv.2305.18703. doi:10.48550/ARXIV.2305.18703. arXiv:2305.18703.

[7] D. Benavides, A. Felfernig, J. A. Galindo, F. Reinfrank, Automated analysis in feature modelling and product configuration, in: Safe and Secure Software Reuse: 13th International Conference on Software Reuse, ICSR 2013, Pisa, June 18-20. Proceedings 13, Springer, 2013, pp. 160–175.

[8] N. Nethercote, P. J. Stuckey, R. Becket, S. Brand, G. J. Duck, G. Tack, MiniZinc: Towards a standard CP modelling language, in: CP, volume 4741 of LNCS, Springer, 2007, pp. 529–543.

[9] A. A. Falkner, A. Haselböck, G. Krames, G. Schenner, R. Taupe, Constraint solver requirements for interactive configuration, in: L. Hotz, M. Aldanondo, T. Krebs (Eds.), Proceedings of the 21st Configuration Workshop, Hamburg, Germany, September 19-20, 2019, volume 2467 of CEUR Workshop Proceedings, CEUR-WS.org, 2019, pp. 65–72. URL: http://ceur-ws.org/Vol-2467/paper-12.pdf.

[10] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, D. Amodei, Language models are few-shot learners, in: H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, H. Lin (Eds.), Advances in Neural Information Processing Systems, volume 33, Curran Associates, Inc., 2020, pp. 1877–1901. URL: https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.

[11] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, D. Zhou, Chain-of-thought prompting elicits reasoning in large language models, in: Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Curran Associates Inc., Red Hook, NY, USA, 2024.

[12] B. Wang, Z. Wang, X. Wang, Y. Cao, R. A. Saurous, Y. Kim, Grammar prompting for domain-specific language generation with large language models, in: A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, S. Levine (Eds.), Advances in Neural Information Processing Systems, volume 36, Curran Associates, Inc., 2023, pp. 65030–65055. URL: https://proceedings.neurips.cc/paper_files/paper/2023/file/cd40d0d65bfebb894ccc9ea822b47fa8-Paper-Conference.pdf.

[13] G. Poesia, A. Polozov, V. Le, A. Tiwari, G. Soares, C. Meek, S. Gulwani, Synchromesh: Reliable code generation from pre-trained language models, in: The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, OpenReview.net, 2022.

[14] L. Pan, A. Albalak, X. Wang, W. Wang, LogicLM: Empowering large language models with symbolic solvers for faithful logical reasoning, in: H. Bouamor, J. Pino, K. Bali (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Association for Computational Linguistics, Singapore, 2023, pp. 3806–3824. URL: https://aclanthology.org/2023.findings-emnlp.248. doi:10.18653/v1/2023.findings-emnlp.248.

[15] P. Kogler, A. Falkner, S. Sperl, Reliable generation of formal specifications using large language models, in: SE 2024 - Companion, Gesellschaft für Informatik e.V., 2024, pp. 141–153. doi:10.18420/sw2024-ws_10.

[16] B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. C. Ferrer, A. Grattafiori, W. Xiong, A. Défossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scialom, G. Synnaeve, M. Ai, Code llama: Open foundation models for code, 2023. URL: https://github.com/facebookresearch/codellama.

[17] MistralAI, Codestral introduction (2024). URL: https://mistral.ai/news/codestral/.

[18] AI@Meta, Llama 3 model card (2024). URL:
https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md.

[19] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, W. E. Sayed, Mistral 7b, 2023. arXiv:2310.06825.

[20] M. Mendonca, M. Branco, D. Cowan, S.p.l.o.t. - software product lines online tools, In Companion to the 24th ACM SIGPLAN International Conference on Object-Oriented Programming Systems, Languages, and Applications, OOPSLA (2009) 761-762. doi:10.1145/1639950.1640002.

[21] Gecode Team, Gecode: Generic constraint development environment, 2006. Available from http://www.gecode.org.

[22] M. LEVANDOWSKY, D. WINTER, Distance between sets, Nature 234 (1971) 34-35. URL: https://doi.org/10.1038/234034a0. doi:10.1038/234034a0.

[23] A. Falkner, A. Haselböck, G. Krames, G. Schenner, H. Schreiner, R. Comploi-Taupe, Solver requirements for interactive configuration, JOURNAL OF UNIVERSAL COMPUTER SCIENCE 26 (2020) 343-. doi:10.3897/jucs.2020.019.

[24] G. Fleischanderl, G. Friedrich, A. Haselböck, H. Schreiner, M. Stumptner, Configuring large systems using generative constraint satisfaction, IEEE Intelligent Systems 13 (1998) 59-68. URL: https://doi.org/10.1109/5254.708434. doi:10.1109/5254.708434.

[25] K. Czarnecki, S. Helsen, U. W. Eisenecker, Formalizing cardinality-based feature models and their specialization, Software Process: Improvement and Practice 10 (2005) 7-29. URL: https://doi.org/10.1002/spip.213. doi:10.1002/spip.213.