DeepSolution: Boosting Complex Engineering Solution Design via Tree-based Exploration and Bi-point Thinking
Authors: Zhuoqun Li, Haiyang Yu, Xuanang Chen, Hongyu Lin, Yaojie Lu, Fei Huang, Xianpei Han, Yongbin Li, Le Sun
Accepted at: acl25

DeepSolution: Boosting Complex Engineering Solution Design via Tree-based Exploration and Bi-point Thinking

Zhuoqun Li1,2, Haiyang Yu3, Xuanang Chen1, Hongyu Lin1*, Yaojie Lu1, Fei Huang3, Xianpei Han1, Yongbin Li3*, Le Sun1
1Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences
2University of Chinese Academy of Sciences
3Tongyi Lab
{lizhuoqun2021, chenxuanang, hongyu, luyaojie, xianpei, sunle}@iscas.ac.cn
{yifei.yhy, f.huang, shuide.lyb}@alibaba-inc.com

Abstract
Designing solutions for complex engineering challenges is crucial in human production activities. However, previous research in the retrieval-augmented generation (RAG) field has not sufficiently addressed tasks related to the design of complex engineering solutions. To fill this gap, we introduce a new benchmark, SolutionBench, to evaluate a system’s ability to generate complete and feasible solutions for engineering problems with multiple complex constraints. To further advance the design of complex engineering solutions, we propose a novel system, SolutionRAG, that leverages the tree-based exploration and bi-point thinking mechanism to generate reliable solutions. Extensive experimental results demonstrate that SolutionRAG achieves state-of-the-art (SOTA) performance on the SolutionBench, highlighting its potential to enhance the automation and reliability of complex engineering solution design in real-world applications. https://github.com/icip-cas/DeepSolution.

1 Introduction
Designing solutions for complex engineering requirements is a crucial work in human production activities (Ogot and Kremer, 2004; ElMaraghy et al., 2012). These requirements typically include multiple real-world constraints and expect complete and feasible solutions (e.g., Design a safe and efficient hospital construction plan in an area with annual rainfall of 3000 millimeters, expansive soil conditions, and frequent seismic activity). Human experts complete such work by consulting extensive professional knowledge, carefully designing, and strictly deliberating, which require significant time and human resources (Kalogerakis et al., 2010; De Weck et al., 2011). With the continuous development of retrieval-augmented generation (RAG) techniques, the engineering fields anticipate a credible RAG system that can automatically generate reliable solutions for these complex engineering requirements (Yu et al., 2024; Zhou et al., 2024).

Unfortunately, prior works in RAG field do not sufficiently research the complex engineering solution design task. Existing relevant papers mainly focus on Long-form QA or Multi-hop QA (Zhu et al., 2024; Tan et al., 2024), where the questions are integrated or composed of multiple sub-questions and the expected answers are typically assembled knowledge paragraphs or entity fragments. Unlike these tasks, requirements of the complex engineering solution design task involve multiple real-world constraints and demand complete and feasible solutions (Fortus et al., 2005; Jonassen et al., 2006), as shown in Figure 1. Therefore, researching complex engineering solution design based on RAG technology is a valuable gap that needs to be filled.

To fill this gap, we first introduce a new benchmark, SolutionBench, to evaluate whether a system can generate complete and feasible solutions for complex engineering requirements with multiple constraints. Firstly, to ensure the data source’s

![This paper proposes the complex engineering solution design task and a new system that can generate reliable solutions via the bi-point thinking tree.](page_728_370_496_312.png)

Figure 1: This paper proposes the complex engineering solution design task and a new system that can generate reliable solutions via the bi-point thinking tree.
authority, authenticity, and diversity, we collect thousands of engineering reports about solution design from authoritative journals in various engineering domains. Then, to build data that is convenient for testing and evaluation, we refer to the generative information extraction technologies (Lu et al., 2022; Zhang et al., 2025) and employ LLMs to extract useful content from these reports based on a manually formatted template, capturing real-world complex requirements, expert-authored solutions, analytical knowledge used to interpret the requirements, technical knowledge applied in addressing the requirements, and explanations for the expert’s solution design process. Finally, we manually verify and revise the extracted content, merge all knowledge within the same domain into a unified knowledge base, and then harvest a complete benchmark for complex engineering solution design that covers eight engineering domains.

To further advance complex engineering solution design, we propose SolutionRAG, which can generate reliable solutions through tree-based exploration and bi-point thinking. Firstly, the improvement process from suboptimal solutions to reliable solutions is flexible, rather than with a fixed reasoning pattern. Therefore, SolutionRAG conducts the tree-based exploration, where each branch represents a different improvement direction. Secondly, due to the presence of multiple real-world constraints within the requirements, system-generated solutions cannot guarantee the satisfaction of all constraints. Therefore, SolutionRAG employs the bi-point thinking, which alternates between solution designing and reviewing during the tree growth, gradually improving reliability of generated solutions. Finally, to balance inference efficiency and performance, SolutionRAG implements pruning based on node evaluation, which can keep the inference process along the most promising solutions and the most helpful reviewed comments.

In experiments, we evaluate various types of methods on SolutionBench to assess their ability in complex engineering solution design, including deep thinking models without RAG, standard RAG approaches, multi-round iterative RAG methods, and our SolutionRAG. Experimental results show that LLMs relying solely on internal knowledge cannot effectively solve such tasks. Previous RAG methods also fail to generate satisfactory solutions. In contrast, our proposed SolutionRAG proves to be a more advanced approach. The main contributions of this paper can be summarized as follows:

• We construct SolutionBench, which can evaluate a system’s ability for complex engineering solution design from real-world scenarios.

• We propose SolutionRAG, which can boost complex engineering solution design through tree-based exploration and bi-point thinking.

• We conduct extensive experiments, and results show existing methods perform poorly and SolutionRAG is an advanced improvement.

2 SolutionBench

As mentioned above, research on complex engineering solution design tasks has significant value in enhancing the productivity of human society, but previous works in RAG field do not explore this in depth. Therefore, this paper introduces a new benchmark, SolutionBench, which can evaluate a system’s ability to design solutions for complex engineering requirements. Specifically, as illustrated in Figure 2, we first collect engineering technical reports about complex solution design from authoritative journals across various engineering fields. Then, based on manually formatted extraction templates, we use powerful LLMs to implement useful content extraction. Finally, after manually checking and removing redundancy, the extracted content is integrated into a complete benchmark. Here is detailed process of constructing SolutionBench:

2.1 Authoritative Data Source

To ensure the credibility of benchmark, we primarily consider two key factors when determining data sources: the authority and authenticity of data, as well as the diversity of engineering domains.

Authority and Authenticity. In order to ensure the benchmark’s evaluation results can accurately reflect the system’s capabilities under real engineering requirements, it is essential to ensure the data sources come from authoritative experts and real-world scenarios. To this end, we select authoritative journals in engineering fields as data sources, choosing engineering reports that involve complex engineering solution design. The requirements in these reports are derived from real industrial scenarios and provided by industry experts under strict peer review, thus ensuring the authenticity and authority of data sources. The detailed list of used engineering journals is shown in Appendix A.

Domain Diversity. Since the need for complex engineering solution design is urgent in multiple
Figure 2: Illustration of the SolutionBench construction method, which includes collecting technology reports from engineering journals to ensure authority and authenticity, extracting useful content based on a manually formatted template and powerful LLMs, and finally harvesting the benchmark after manual verification and merging.

engineering domains, the data sources used to construct benchmark must cover a broad range of domains to ensure comprehensive evaluation. To this end, we select authoritative journals from eight major categories based on the discipline classification mechanism of the search websites: Environment, Mining, Transportation, Aerospace, Telecom, Architecture, Water Resource, and Farming. The coverage of these fields ensures that the data sources include diverse engineering scenarios, providing a broad reference for system evaluation.

2.2 Template-based Extraction via LLM

To transform original engineering technical reports into data for evaluation and scoring, we format a template manually and extract following content from each report via LLMs: requirement, solution, analytical knowledge, technical knowledge, and explanation, based on the generative information extraction (Lu et al., 2022; Zhang et al., 2025).

Template. In order to facilitate the testing and scoring, we format an extraction template including following keys: (1) Requirement, which refers to the complex needs from real engineering scenarios addressed in reports, (2) Solution, which is the complete and reliable solution designed by top industry experts, (3) Analytical Knowledge, which is the professional knowledge used by experts when analyzing the complex requirements during solution design process (e.g., Impact of earthquakes is mainly reflected in horizontal vibration), (4) Technical Knowledge, which is the professional knowledge used by experts to address the complex requirements and develop the complete solutions (e.g., Nano bearings can reduce the horizontal seismic vibration by special structure), (5) Explanation, which outlines how the experts use analytical knowledge and technical knowledge to analyze the complex requirements and gradually design complete solutions. This explanation can serve as an auxiliary reference during the evaluation process. The complete template used to implement the extraction process is shown in the Appendix B.

Extraction Process. Since the original engineering reports are in PDF format and cannot be directly processed for content extraction, we first use the marker tool1 to convert the PDF files into plain text. And then we input the plain text along with the manually formatted template into GPT-4o (OpenAI, 2024a), extracting content as described in the template. Finally we transform extracted content into JSON format and save it for further process.

2.3 Manual Data Verification

To further ensure the credibility of the benchmark, we manually check correctness and remove the redundancy for the extracted content.

Correctness Checking. Since the LLM is a probabilistic model and cannot guarantee that every extracted piece of content aligns with our specifications, we manually check each extracted content. On one hand, we examine whether the content matches the information in original engineering reports, on the other hand, we assess whether the content adheres to definitions in the template. For incorrect content, we directly correct it manually.

Redundancy Removing. Since we select many technical reports as data sources for each engineering domain, the analytical knowledge and technical knowledge used to address complex requirements from the same domain may be similar or even identical, resulting in redundancy when constructing a large knowledge base. Therefore, we manually check duplicates for the knowledge in each domain. If duplicates are found, we manually merge the redundant knowledge to one knowledge.

1https://github.com/VikParuchuri/marker
<table>
  <tr>
    <th>Engineering Domain</th>
    <th># Datapoint</th>
    <th># Knowledge</th>
  </tr>
  <tr>
    <td>Environment (Env.)</td>
    <td>119</td>
    <td>554</td>
  </tr>
  <tr>
    <td>Mining (Min.)</td>
    <td>117</td>
    <td>543</td>
  </tr>
  <tr>
    <td>Transportation (Tra.)</td>
    <td>124</td>
    <td>870</td>
  </tr>
  <tr>
    <td>Aerospace (Aer.)</td>
    <td>115</td>
    <td>802</td>
  </tr>
  <tr>
    <td>Telecom (Tel.)</td>
    <td>116</td>
    <td>840</td>
  </tr>
  <tr>
    <td>Architecture (Arc.)</td>
    <td>118</td>
    <td>858</td>
  </tr>
  <tr>
    <td>Water Resource (Wat.)</td>
    <td>119</td>
    <td>802</td>
  </tr>
  <tr>
    <td>Farming (Far.)</td>
    <td>122</td>
    <td>868</td>
  </tr>
</table>

Table 1: Statistics of the SolutionBench, which include data and knowledge across eight engineering domains. The number of datapoints in dataset and the number of knowledge in knowledge base are shown above.

2.4 Datapoint and Knowledge Base

After above manual verification, we do content integrate and get 8 high-quality datasets for the 8 domains, correspondingly with 8 knowledge base. The detailed statistics of benchmark is in Table 1.

Datapoint Format. The content of datapoints of every domain is as following formula:

\[
\mathcal{D} = \{q_i, s_i, \{k_j^{(a)}\}_{j=1}^{A_i}, \{k_j^{(t)}\}_{j=1}^{T_i}, e_i\}_{i=1}^N
\]

where \( \mathcal{D} \) is the dataset for one domain, \( N \) is data number, \( q_i \) is one requirement, \( s_i \) is the goldn solution, \( k_j^{(a)} \) is an analytical knowledge used for \( q_i \) and \( A_i \) is the total number, \( k_j^{(t)} \) is a technical knowledge used for \( q_i \) and \( T_i \) is the total number.

Knowledge Base. In order to get the referential knowledge base for each engineering domain, we collect all the \( k_j^{(a)} \) and \( k_j^{(t)} \) within the same domain into a large corpus, as shown in following:

\[
\mathcal{K} = \cup[\{k_j^{(a)}\}_{j=1}^{A_i}, \{k_j^{(t)}\}_{j=1}^{T_i}] = \{k_i\}_{i=1}^M
\]

where \( \mathcal{K} \) is the knowledge base for one domain, and \( M \) is the number of knowledge in \( \mathcal{K} \).

Evaluation Formulating. There are two ways to using SolutionBench for evaluation. The first one is that given a requirement \( q \) and expect an reliable solution \( \hat{s} \), as shown in following formula:

\[
\hat{s} = \mathcal{F}(q)
\]

And the second one is RAG setting, which extra provides the relevant knowledge base for retrieval and augmentation, as shown in following formula:

\[
\hat{s} = \mathcal{F}(q, \mathcal{K})
\]

Since the completion of above tasks requires various engineering expertise, which is prone to hallucination issues in regular-sized LLMs (Jiang et al., 2023), we mainly focus on the RAG setting in this paper. At the same time, we also test some powerful deep reasoning LLMs in experiments without using RAG, the details are in Section 4.

3 SolutionRAG

To further advance research in complex engineering solution design, we propose SolutionRAG, a system that can generate reliable solutions through tree-based exploration and bi-point thinking. Specially, as illustrated in Figure 3, since the improvement process from a suboptimal solution to a reliable one is flexible and lacks a fixed reasoning pattern, SolutionRAG performs tree-base exploration to find the most effective improvement process for each input requirement. Moreover, due to the multiple real-world constraints within the requirements, the system cannot directly guarantee the generated solutions satisfy all constraints. Therefore, SolutionRAG employs a bi-point thinking approach, alternating between solution design and review, gradually enhancing the solution’s completeness and reliability. Finally, to balance inference performance and efficiency, SolutionRAG employs node evaluation to prune the tree, ensuring that the inference process follows the most promising solutions and the most helpful reviewed comments.

3.1 Bi-point Thinking Tree

To explore the optimal process for solution improvement during inference and ensure the output solutions meet all constraints in the requirements, SolutionRAG performs inference based on a bi-point thinking tree, which consists of alternating connected solution nodes and comment nodes.

Solution Node. The content within a solution nodes is the solution designed for the given requirement, which is expected to be a complete and feasible solution meeting all constraints specified in the requirement. The solution nodes at the shallower levels of the tree typically have a lower degree of reliability for the given requirement, while those at deeper levels have a higher degree of reliability. For convenience, we use \( s_j^{(i)} \) represents the \( j \)-th solution node at the \( i \)-th layer of the tree.

Comment Node. The content within a comment node is the comment obtained from reviewing a
Figure 3: Illustration of SolutionRAG, we set the child number of each node as 2 for easy presentation above. SolutionRAG uses tree-based exploration to find optimal solution improvement process, bi-point thinking to guarantee generated solutions satisfy all constraints, and a pruning mechanism to balance efficiency and performance.

particular solution, which indicates the aspects in which the solution still has deficiencies with respect to the given requirement. For convenience of description, we use \( c_j^{(i+1)} \) represents the j-th comment node at the \((i+1)\)-th layer of the tree.

Tree Structure. The above-mentioned solution nodes and comment nodes are alternately connected to form a bi-point thinking tree, where the child nodes of a solution node are comment nodes, and the child nodes of a comment node are solution nodes, as shown in the following formula:

\[
s_j^{(i)} \rightarrow \{c_j^{(i+1)}\}_{h=1}^H \tag{5}
\]
\[
c_j^{(i+1)} \rightarrow \{s_h^{(i+2)}\}_{h=1}^H \tag{6}
\]

where \( H \) is the number of child node in tree. The content of the root node of the tree is the requirement \( q \), so \( i \) is at least one in above formula.

3.2 Solution Improvement via Tree Growth

In this section, we introduce how SolutionRAG specifically achieves continuous improvement of solutions through the growth process of aforementioned bi-point thinking tree, including the node expansion and node evaluation process.

Node Expansion. During the growth process of the bi-point thinking tree, there are two types of node expansion actions, one is based on the requirement or comment node to design new solution nodes, and the other is based on reviewing the solution node to create new comment nodes.

(1) Design. Given the requirement \( q \) and a specific comment \( c_j^{(i+1)} \) as input (if \( i \) is at least one), the design process generate \( H \) proposals \( \{p_h\}_{h=1}^H \) through random sampling using a LLM, representing \( H \) different directions for designing:
\[
\{p_h\}_{h=1}^H = \mathrm{LLM}(q, c_j^{(i+1)}) \tag{7}
\]

Then, small-scale relevant knowledge \( \mathcal{K}_h \) is retrieved from the knowledge base \( \mathcal{K} \) for each \( p_h \):
\[
\mathcal{K}_h = \mathrm{Retrieval}(p_h, \mathcal{K}) = \{k_r\}_{r=1}^R \tag{8}
\]

Finally, \( q, c_j^{(i+1)}, \mathcal{K}_h \), and the history solution \( s_z^{(i)} \) are concatenated as input, allowing the LLM to output a more refined new solution:
\[
s_h^{(i+2)} = \mathrm{LLM}(q, s_z^{(i)}, c_j^{(i+1)}, \mathcal{K}_h) \tag{9}
\]

Thus, we obtain \( H \) new solutions \( \{s_h^{(i+2)}\}_{h=1}^H \) refined based on the comment \( c_j^{(i+1)} \). Note that during the generation of solution nodes in the first layer, there are no previous solutions or comments, so we initialize \( s_z^{(i)} \) and \( c_j^{(i+1)} \) as empty text.

(2) Review. Similar to the previous process, the review process also consists of three steps: First, proposals \( \{p_h\}_{h=1}^H \) are generated based on \( q \) and \( s_z^{(i)} \), representing \( H \) distinct review directions. Next, knowledge \( \mathcal{K}_h \) is retrieved for each \( p_h \). Finally, comments \( \{c_h^{(i+1)}\}_{h=1}^H \) are generated for \( s_z^{(i)} \) based on \( q, s_z^{(i)} \), and \( \mathcal{K}_h \). The maximum depth of the bi-point thinking tree, denoted as \( L \), is a hyper-parameter. Prompts for this part are Appendix C.

Node Evaluation. As described in above node expansion part, the number of nodes becomes enormous as the tree grows, leading to significant time
consumption during inference. To this end, during tree growth, we do prune by each node score from its child nodes, meaning whether \( s_j^{(i)} \) is an reliable solution based on \( \{c_h^{(i+1)}\}_{h=1}^H \) and whether \( c_j^{(i+1)} \) is a helpful comment for solution improvement based on \( \{s_h^{(i+2)}\}_{h=1}^H \). Specifically, for judging \( s_j^{(i)} \), we put \( s_j^{(i)}, c_j^{(i+1)} \), and a suffix \( u_s \) together as the LLM input, and get the reliability score \( \mathcal{J}_h(s_j^{(i)}) \) by calculating LLM-predicted average logits of \( u_s \):

\[
\mathcal{J}_h(s_j^{(i)}) = \mathrm{Logits}(u_s|s_j^{(i)}, c_j^{(i+1)})
\]

where \( u_s \) is “According to the comment, above solution is reliable”. And then get final score \( \mathcal{J}(s_j^{(i)}) \) for \( s_j^{(i)} \) by average all \( \{\mathcal{J}_h(s_j^{(i)})\}_{h=1}^H \). Similarly, for judging \( c_j^{(i+1)} \), we use \( s_z^{(i)}, c_j^{(i+1)}, s_h^{(i+2)} \), and \( u_c \) as input, and get the helpfulness score \( \mathcal{J}_h(c_j^{(i+1)}) \) by calculating LLM-predicted average logits:

\[
\mathcal{J}_h(c_j^{(i+1)}) = \mathrm{Logits}(u_c|s_z^{(i)}, c_j^{(i+1)}, s_h^{(i+2)})
\]

where \( u_c \) is “Comparing the new solution and old solution, the comment is helpful”, and get \( \mathcal{J}(c_j^{(i+1)}) \) after same averaging process.

During the tree growth, for each layer we only keep the \( W \) highest-scoring nodes, aiming to keep the inference process always focus on the most promising solutions and the most helpful reviewed comments, thus achieving a balance between efficiency and performance. The nodes that are used in final inference process are called retained nodes, while those not-used are pruned nodes.

4 Experiments

4.1 Experimental Settings

Evaluation Metrics. Since expected system output in SolutionBench are solutions that may have various textual expressions, rule-based metrics are difficult to provide a score that aligns with human habits (Xu et al., 2023; Mayfield et al., 2024). To this end, we follow metrics of previous Long-form QA evaluation methods (Tan et al., 2024; Wang et al., 2024a), using GPT-4o2 as score evaluator to compute two scores, (1) Analytical Score: We integrate the expert-designed solution, analytical knowledge used by experts, and the explanation as reference, and then let evaluator judge whether the system-designed solution, like the expert-designed one, uses the correct analytical knowledge to adequately analysis the complex constraints in requirements. (2) Technical Score: Similarly, we integrate the expert-designed solution, technical knowledge used by experts, and the explanation as reference, and then let evaluator judge whether the system-designed solution, like the expert-designed one, uses the correct technical knowledge to tackle the complex constraints in requirements. Both analytical score and technical score are range from 0 to 100. Prompts for this part are in Appendix D.

Selected Baselines. In order to comprehensively evaluate the abilities of various types of systems in solving complex engineering solution design tasks, we extensively select multiple types of methods as baselines in the experiments. Specifically, (1) Deep Reasoning Models: This type includes models like o1-2024-12-17 (OpenAI, 2024b), GLM-Zero-Preview (Zhipu, 2024), and QwQ-32B-Preview (Qwen, 2024), which possess strong long-chain reasoning capabilities, but do not utilize external knowledge like RAG. (2) Single-round RAG Methods: These methods perform only one round of retrieval and generation, where Naive-RAG (Lewis et al., 2020) does not process the retrieval results, while Rerank-RAG (Li et al., 2023) uses an additional model to re-rank the retrieval results. (3) Multi-round RAG Methods: These methods conduct multiple rounds of RAG, iteratively performing tasks such as question rewriting, retrieval, filtering, and generating intermediate answer. We choose 3 accredited methods, which are Self-RAG (Asai et al., 2024), GenGround (Shi et al., 2024), and RQ-RAG (Chan et al., 2024).

Implementation Details. For deep reasoning models in baselines, we directly use official API for experiments. For the single-round and multi-round RAG methods, we follow their official process. For SolutionRAG, we set maximum tree depth \( L \) as 5, number of child per node \( H \) as 2, and number of retained node \( W \) in pruning as 1. To ensure fair comparison, we adopt the following same implementation setting for SolutionRAG and all RAG-based methods in baselines: base model is Qwen2.5-TB-Instruct (Team, 2024), retrieval model is NV-Embed-v2 (Lee et al., 2025), and the number of retrieval results \( R \) is setting as 10. For convenience, in all RAG-based experiments, we deploy the base model as API by vLLM3.

2https://openai.com/index/hello-gpt-4o/
3https://pypi.org/project/vllm/
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="4">Env.</th>
    <th colspan="4">Min.</th>
    <th colspan="4">Tra.</th>
    <th colspan="4">Aer.</th>
    <th colspan="4">Tel.</th>
    <th colspan="4">Arc.</th>
    <th colspan="4">Wat.</th>
    <th colspan="4">Far.</th>
  </tr>
  <tr>
    <th>AS</th><th>TS</th><th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th><th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th><th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th><th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th><th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th><th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th><th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th><th>AS</th><th>TS</th>
  </tr>
  <tr>
    <td colspan="26"><b>Deep Reasoning Models</b></td>
  </tr>
  <tr>
    <td>ol-2024-12-17 (OpenAI, 2024b)</td><td>60.5</td><td>48.3</td><td>51.9</td><td>37.5</td><td>57.3</td><td>44.7</td><td>57.8</td><td>47.6</td><td>63.5</td><td>52.3</td><td>61.2</td><td>52.0</td><td>59.9</td><td>50.4</td><td>62.9</td><td>52.2</td>
  </tr>
  <tr>
    <td>GLM-Zero-Preview (Zhipu, 2024)</td><td>47.0</td><td>30.6</td><td>43.2</td><td>22.2</td><td>45.2</td><td>27.0</td><td>42.3</td><td>25.7</td><td>45.1</td><td>31.7</td><td>47.7</td><td>32.4</td><td>47.3</td><td>30.8</td><td>51.4</td><td>36.6</td>
  </tr>
  <tr>
    <td>QwQ-32B-Preview (Qwen, 2024)</td><td>54.3</td><td>38.7</td><td>48.0</td><td>27.9</td><td>47.2</td><td>29.3</td><td>47.4</td><td>31.9</td><td>52.2</td><td>35.9</td><td>51.3</td><td>35.6</td><td>49.2</td><td>33.0</td><td>53.4</td><td>37.0</td>
  </tr>
  <tr>
    <td colspan="26"><b>Single-round RAG Methods</b></td>
  </tr>
  <tr>
    <td>Naïve-RAG (Lewis et al., 2020)</td><td>64.8</td><td>62.2</td><td>57.2</td><td>40.1</td><td>62.7</td><td>54.9</td><td>67.7</td><td>65.4</td><td>67.4</td><td>66.8</td><td>66.2</td><td>63.3</td><td>66.0</td><td>57.5</td><td>65.7</td><td>63.0</td>
  </tr>
  <tr>
    <td>Rerank-RAG (Li et al., 2023)</td><td>62.7</td><td>60.7</td><td>53.4</td><td>38.4</td><td>60.0</td><td>49.7</td><td>65.6</td><td>65.2</td><td>66.1</td><td>63.4</td><td>66.4</td><td>62.8</td><td>64.1</td><td>55.4</td><td>64.0</td><td>59.7</td>
  </tr>
  <tr>
    <td colspan="26"><b>Multi-round RAG Methods</b></td>
  </tr>
  <tr>
    <td>Self-RAG (Asai et al., 2024)</td><td>64.2</td><td>63.6</td><td>56.1</td><td>41.6</td><td>62.9</td><td>56.5</td><td>68.8</td><td>69.9</td><td>67.6</td><td>66.9</td><td>66.7</td><td>65.9</td><td>64.8</td><td>58.6</td><td>65.1</td><td>61.1</td>
  </tr>
  <tr>
    <td>GenGround (Shi et al., 2024)</td><td>54.8</td><td>46.1</td><td>53.0</td><td>33.3</td><td>54.7</td><td>37.2</td><td>55.7</td><td>46.0</td><td>58.3</td><td>50.7</td><td>60.1</td><td>50.7</td><td>60.4</td><td>48.9</td><td>59.8</td><td>52.7</td>
  </tr>
  <tr>
    <td>RQ-RAG (Chan et al., 2024)</td><td>53.5</td><td>44.4</td><td>48.9</td><td>28.7</td><td>53.8</td><td>38.8</td><td>55.0</td><td>46.1</td><td>57.9</td><td>44.6</td><td>56.3</td><td>46.9</td><td>54.3</td><td>39.8</td><td>57.2</td><td>45.2</td>
  </tr>
  <tr>
    <td colspan="26"><b>Tree-based Exploration and Bi-point Thinking</b></td>
  </tr>
  <tr>
    <td>SolutionRAG (Ours)</td><td>66.4</td><td>67.9</td><td>59.7</td><td>50.5</td><td>64.1</td><td>58.5</td><td>69.9</td><td>72.7</td><td>68.8</td><td>69.0</td><td>67.9</td><td>68.0</td><td>66.0</td><td>60.7</td><td>66.9</td><td>65.2</td>
  </tr>
</table>

Table 2: Main experimental results on SolutionBench with eight engineering domains, the AS is the analytical score and TS is the technical score. The table shows that previous methods perform poorly for complex engineering solution design. In contrast, our SolutionRAG is able to output more complete and reliable solutions.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="4">Env.</th>
    <th colspan="4">Min.</th>
    <th colspan="4">Tra.</th>
    <th colspan="4">Aer.</th>
    <th colspan="4">Tel.</th>
    <th colspan="4">Arc.</th>
    <th colspan="4">Wat.</th>
    <th colspan="4">Far.</th>
    <th colspan="4">Overall</th>
  </tr>
  <tr>
    <th>AS</th><th>TS</th><th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th><th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th><th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th><th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th><th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th><th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th><th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th><th>AS</th><th>TS</th>
  </tr>
  <tr>
    <td>SolutionRAG</td><td>66.4</td><td>67.9</td><td>59.7</td><td>50.5</td><td>64.1</td><td>58.5</td><td>69.9</td><td>72.7</td><td>68.8</td><td>69.0</td><td>67.9</td><td>68.0</td><td>66.0</td><td>60.7</td><td>66.9</td><td>65.2</td><td>66.2</td><td>64.1</td>
  </tr>
  <tr>
    <td>w/o tree structure</td><td>63.5</td><td>66.5</td><td>57.3</td><td>46.2</td><td>63.1</td><td>57.4</td><td>60.8</td><td>68.4</td><td>60.9</td><td>63.7</td><td>66.2</td><td>67.2</td><td>65.6</td><td>59.9</td><td>64.2</td><td>63.9</td><td>62.7</td><td>61.7</td>
  </tr>
  <tr>
    <td>w/o bi-point thinking</td><td>62.8</td><td>64.7</td><td>55.6</td><td>47.3</td><td>61.5</td><td>55.7</td><td>63.2</td><td>68.3</td><td>62.6</td><td>64.8</td><td>67.5</td><td>67.3</td><td>64.4</td><td>59.1</td><td>65.2</td><td>64.7</td><td>62.9</td><td>61.5</td>
  </tr>
</table>

Table 3: Ablation results for tree-based exploration and bi-point thinking. The table shows that both mechanisms have obviously positive effects for SolutionRAG and exhibit a similar level of importance in the overall.

4.2 Overall Results

Results compared with baselines are shown in the Table 2, there are two main conclusions:

Previous methods fail to effectively address the complex engineering solution design. The table shows that, on one hand, deep reasoning models without RAG perform poorly across all eight domains in SolutionBench. For example, GLM-Zero-Preview achieves an analytical score of only 42.3 in the aerospace domain. On the other hand, RAG-based methods achieve some better performance but still remain at relatively low levels. For instance, Naïve-RAG obtains a technical score of only 40.1 in the mining engineering domain, and Self-RAG achieves a technical score of just 63.6 in the environmental engineering domain.

In contrast, SolutionRAG is an effective system for complex engineering solution design tasks. The table shows that SolutionRAG achieves SOTA performance across all of eight domains in the benchmark, demonstrating a significant improvement over baseline methods. For example, in the mining domain, SolutionRAG improves the technical score by 10.4 compared to Naïve-RAG and by 8.9 compared to Self-RAG. These experimental results confirm that SolutionRAG can effectively handle complex solution design tasks in various real-world engineering scenarios.

4.3 Ablation Results

Since tree-based exploration and bi-point thinking are two key mechanisms in SolutionRAG, we conduct two ablation experiments, results are shown in Table 3, where “w/o tree structure” is that each node generates only one child, resulting in a single-chain inference pattern, and “w/o bi-point thinking” is that the tree does not include reviewing and all nodes are solutions, leading to a uni-point thinking inference pattern. There are two main conclusions:

Both of the tree-based exploration and bi-point thinking have positive effects. The table shows that removing either mechanism leads to a significant decline in performance, indicating that these two mechanisms are indeed central to solving complex engineering solution design tasks.

Tree-based exploration and bi-point thinking exhibit a similar level of importance. The table shows that after removing these two mechanisms, overall performance decline is quite similar, indicating these two mechanisms hold a comparable level of importance in SolutionRAG.
Figure 4: Performance changes during the tree growth. The figure shows that scores become higher as the tree grows, proving SolutionRAG can indeed improve the solution scores as inference being deep.

Figure 5: Effectiveness of node evaluation mechanism. The figure shows that scores in retained nodes are higher than in pruned nodes, thus the node evaluation is an effective method for judging and pruning in SolutionRAG.

4.4 Detailed Analysis

In order to further validate the SolutionRAG, we do some detailed analysis, including performance changing during the tree growth process and effectiveness of the node evaluation in SolutionRAG.

Performance during Tree Growth. To examine whether the solutions actually improve as the tree depth increases in SolutionRAG inference, we score the solutions from the layer-1, 3, and 5 of the tree. The experimental results are shown in Figure 4, performance gradually improves from the shallow layer to the deep layer, which proves that SolutionRAG can indeed improve the solution as inference process being deep.

Effectiveness of Node Evaluation. To examine whether node evaluation mechanism for pruning the tree is effective, we compare the scores of solutions from the retained nodes with those from the pruned nodes. The results are shown in Figure 5, where the scores of solutions from the retained nodes are significantly higher than pruned nodes, which proves that node evaluation is an effective mechanism for judging and pruning.

5 Related Work

Complex QA Tasks. Recent works in the RAG field mainly focused on knowledge-based question answering tasks that require some level of reasoning. (1) Multi-hop QA. The question is a combination of multiple sub-questions, and the expected answer is an entity fragment from relevant knowledge documents (Yang et al., 2018; Ho et al., 2020; Zhu et al., 2024; Wu et al., 2024). (2) Long-form QA. The question is an open-ended and comprehensive question, and the expected answer is a text paragraph formed by integrating knowledge fragments from relevant documents (Fan et al., 2019; Stelmakh et al., 2022; Tan et al., 2024; Qi et al., 2024). Compared to above two tasks, questions of complex engineering solution design are with multiple real-world constraints. And the expected answer is a solution needing flexible improvement process, rather than an entity fragment or simply integrated paragraph. Therefore, complex engineering solution design is a novel and challenging task.

Advanced RAG. Prior advanced RAG systems use a multi-round approach to iteratively perform rewriting, retrieval, reranking, and generating intermediate answers (Asai et al., 2024; Shi et al., 2024; Chan et al., 2024; Wang et al., 2024b; Tran et al., 2024; Yu et al., 2024). Compared to these systems, SolutionRAG is with a bi-point thinking tree, which can respond to challenges of complex engineering solution design. Recently some papers construct RAG systems based on MCTS, achieving better performance through deep thinking (Jiang et al., 2024; Li et al., 2025a; Wu et al., 2025). However, these methods lack a mechanism to ensure that all engineering requirements are met, thus failing to guarantee the reliability of solutions.

6 Conclusion

In this paper, we first construct SolutionBench based on engineering reports across various domains, which can examine the ability of systems on complex engineering solution design. Further, we propose SolutionRAG, which explore the optimal solution-improvement process and gradually generates reliable solutions by a bi-point thinking tree. In experiments, previous methods perform poorly in complex engineering solution design task, while SolutionRAG represents a good improvement over existing approaches. This paper offers a promising direction and can inspire the further research.
7 Limitations

Complex engineering solution design is a task requiring deep research based on professional knowledge, which demands the model has strong capabilities in problem analysis, solution reasoning, and critical thinking. In this paper, due to limited GPU computational resources, we construct the system by the existing capabilities of LLMs, without considering special training. Therefore, a possible direction for future work is to use reinforcement learning to train LLMs, in order to develop more powerful complex engineering solution design systems. Additionally, due to the same limitation in GPU computational resources, we do not extensively explore hyperparameters such as the width and depth of the tree in our experiments. This could be a valuable research topic for future work.

Acknowledgments

We sincerely thank the reviewers for their insightful comments and valuable suggestions. This work was supported by Beijing Municipal Science and Technology Project (Nos. Z231100010323002), Beijing Natural Science Foundation (L243006), the Natural Science Foundation of China (No. 62306303, 62476265) and the Basic Research Program of ISCAS (Grant No. ISCAS-ZD-202401).

References

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations.

Chi-Min Chan, Chunpu Xu, Ruibin Yuan, Hongyin Luo, Wei Xue, Yike Guo, and Jie Fu. 2024. RQ-RAG: Learning to refine queries for retrieval augmented generation. In First Conference on Language Modeling.

Olivier L De Weck, Daniel Roos, and Christopher L Magee. 2011. Engineering systems: Meeting human needs in a complex technological world. Mit Press.

Waguih ElMaraghy, Hoda ElMaraghy, Tetsuo Tomiyama, and Laszlo Monostori. 2012. Complexity in engineering design and manufacturing. CIRP annals, 61(2):793–814.

Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558–3567, Florence, Italy. Association for Computational Linguistics.

David Fortus, Joseph Krajcik, Ralph Charles Dershimer, Ronald W Marx, and Rachel Mamlok-Naaman. 2005. Design-based science and real-world problem-solving. International Journal of Science Education, 27(7):855–879.

Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625, Barcelona, Spain (Online). International Committee on Computational Linguistics.

Jinhao Jiang, Jiayi Chen, Junyi Li, Ruiyang Ren, Shijie Wang, Wayne Xin Zhao, Yang Song, and Tao Zhang. 2024. Rag-star: Enhancing deliberative reasoning with retrieval augmented verification and refinement. Preprint, arXiv:2412.12881.

Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969–7992, Singapore. Association for Computational Linguistics.

David Jonassen, Johannes Strobel, and Chwee Beng Lee. 2006. Everyday problem solving in engineering: Lessons for engineering educators. Journal of engineering education, 95(2):139–151.

Katharina Kalogerakis, Christian Lüthje, and Cornelius Herstatt. 2010. Developing innovations based on analogies: experience from design and engineering consultants. Journal of Product Innovation Management, 27(3):418–436.

Chanky Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2025. Nv-embed: Improved techniques for training llms as generalist embedding models. Preprint, arXiv:2405.17428.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474. Curran Associates, Inc.

Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. 2025a. Search-o1: Agentic search-enhanced large reasoning models. Preprint, arXiv:2501.05366.

Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. Preprint, arXiv:2308.03281.

Zhuoqun Li, Xuanang Chen, Haiyang Yu, Hongyu Lin, Yaojie Lu, Qiaoyu Tang, Fei Huang, Xianpei
Han, Le Sun, and Yongbin Li. 2025b. StructRAG: Boosting knowledge intensive reasoning of LLMs via inference-time hybrid information structurization. In The Thirteenth International Conference on Learning Representations.

Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2022. Unified structure generation for universal information extraction. Preprint, arXiv:2203.12277.

James Mayfield, Eugene Yang, Dawn Lawrie, Sean MacAvaney, Paul McNamee, Douglas W. Oard, Luca Soldaini, Ian Soboroff, Orion Weller, Efsun Kayi, Kate Sanders, Marc Mason, and Noah Hibbler. 2024. On the evaluation of machine-generated reports. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, page 1904–1915, New York, NY, USA. Association for Computing Machinery.

Madara Ogot and Gul Kremer. 2004. Engineering design: a practical guide. Trafford Publishing.

OpenAI. 2024a. Gpt-4o system card. Preprint, arXiv:2410.21276.

OpenAI. 2024b. Openai o1 system card. Preprint, arXiv:2412.16720.

Zehan Qi, Rongwu Xu, Zhijiang Guo, Cuxiang Wang, Hao Zhang, and Wei Xu. 2024. long*rag: Evaluating long-context & long-form retrieval-augmented generation with key point recall. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4852–4872, Miami, Florida, USA. Association for Computational Linguistics.

Team Qwen. 2024. Qwq: Reflect deeply on the boundaries of the unknown.

Zhengliang Shi, Shuo Zhang, Weiwei Sun, Shen Gao, Pengjie Ren, Zhumin Chen, and Zhaochun Ren. 2024. Generate-then-ground in retrieval-augmented generation for multi-hop question answering. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7339–7353, Bangkok, Thailand. Association for Computational Linguistics.

Ivan Stelmakh, Yi Luan, Bhuvan Dhingra, and Ming-Wei Chang. 2022. ASQA: Factoid questions meet long-form answers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8273–8288, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Haochen Tan, Zhijiang Guo, Zhan Shi, Lu Xu, Zhili Liu, Yunlong Feng, Xiaoguang Li, Yasheng Wang, Lifeng Shang, Qun Liu, and Linqi Song. 2024. ProxyQA: An alternative framework for evaluating long-form text generation with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6806–6827, Bangkok, Thailand. Association for Computational Linguistics.

Qwen Team. 2024. Qwen2.5: A party of foundation models.

Hieu Tran, Zonghai Yao, Junda Wang, Yifan Zhang, Zhichao Yang, and Hong Yu. 2024. Rare: Retrieval-augmented reasoning enhancement for large language models. Preprint, arXiv:2412.02830.

Minzheng Wang, Longze Chen, Cheng Fu, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, and Yongbin Li. 2024a. Leave no document behind: Benchmarking long-context llms with extended multi-doc qa. Preprint, arXiv:2406.17419.

Ruobing Wang, Daren Zha, Shi Yu, Qingfei Zhao, Yuxuan Chen, Yixuan Wang, Shuo Wang, Yukun Yan, Zhenghao Liu, Xu Han, Zhiyuan Liu, and Maosong Sun. 2024b. Retriever-and-memory: Towards adaptive note-enhanced retrieval-augmented generation. Preprint, arXiv:2410.08821.

Feijie Wu, Zitao Li, Fei Wei, Yaliang Li, Bolin Ding, and Jing Gao. 2025. Talk to right specialists: Routing and planning in multi-agent system for question answering. Preprint, arXiv:2501.07813.

Jian Wu, Linyi Yang, Zhen Wang, Manabu Okumura, and Yue Zhang. 2024. Coqa: A step-wise counterfactual multi-hop qa benchmark. Preprint, arXiv:2402.11924.

Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi. 2023. A critical evaluation of evaluations for long-form question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3225–3245, Toronto, Canada. Association for Computational Linguistics.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. Preprint, arXiv:1809.09600.

Tian Yu, Shaolei Zhang, and Yang Feng. 2024. Auto-rag: Autonomous retrieval-augmented generation for large language models. arXiv preprint arXiv:2411.19443.

Zikang Zhang, Wangjie You, Tianci Wu, Xinrui Wang, Juntao Li, and Min Zhang. 2025. A survey of generative information extraction. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4840–4870, Abu Dhabi, UAE. Association for Computational Linguistics.

Zhipu. 2024. Glim-zero.

Yujia Zhou, Zheng Liu, and Zhicheng Dou. 2024. Boosting the potential of large language models with an intelligent information assistant. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
Andrew Zhu, Alyssa Hwang, Liam Dugan, and Chris Callison-Burch. 2024. Fanoutqa: A multi-hop, multi-document question answering benchmark for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 18–37.

A List of Engineering Journals

In order to ensure that the data sources used to construct the benchmark are authentic, authoritative, and diverse, we select engineering reports on solution design from authoritative journals in multiple engineering fields as our data sources. If the report is in Chinese, we extract the useful content, then use GPT-4o to translate the content into English and manually verify its accuracy. We list the used engineering journals, including the journal name and ISSN meaning the international standard serial number. The detailed list of used engineering journals is shown in Table 4, 9 and 10.

<table>
  <tr>
    <th>Journal Name</th>
    <th>Environment</th>
    <th>ISSN</th>
  </tr>
  <tr>
    <td>Journal of Environmental Engineering Technology</td>
    <td></td>
    <td>1674-991X</td>
  </tr>
  <tr>
    <td>Environmental Sanitation Engineering</td>
    <td></td>
    <td>1005-8206</td>
  </tr>
  <tr>
    <td>The Administration and Technique of Environmental Monitoring</td>
    <td></td>
    <td>1006-2019</td>
  </tr>
  <tr>
    <td>Environmental and Development</td>
    <td></td>
    <td>2095-672X</td>
  </tr>
  <tr>
    <td>Environmental Protection and Technology</td>
    <td></td>
    <td>1674-0254</td>
  </tr>
  <tr>
    <td>Green Environmental Protection Building Materials</td>
    <td></td>
    <td>1673-6680</td>
  </tr>
  <tr>
    <td>Journal of Modern Technology of Urban Construction</td>
    <td></td>
    <td>1674-7046</td>
  </tr>
  <tr>
    <td>Urban Management and Science & Technology</td>
    <td></td>
    <td>1008-2271</td>
  </tr>
  <tr>
    <td>Science and Technology Square</td>
    <td></td>
    <td>1671-4792</td>
  </tr>
  <tr>
    <td>Construction Materials & Decoration</td>
    <td></td>
    <td>1673-0038</td>
  </tr>
  <tr>
    <td>Industrial Engineering</td>
    <td></td>
    <td>2096-1182</td>
  </tr>
  <tr>
    <td>Instrument Standardization & Metrology</td>
    <td></td>
    <td>1672-5611</td>
  </tr>
  <tr>
    <td>Northwest Hydropower</td>
    <td></td>
    <td>1006-2610</td>
  </tr>
  <tr>
    <td>Technology & Economics in Petrochemicals</td>
    <td></td>
    <td>1674-1099</td>
  </tr>
  <tr>
    <td>Water Resources & Technology</td>
    <td></td>
    <td>1009-0177</td>
  </tr>
  <tr>
    <td>Construction Science and Technology</td>
    <td></td>
    <td>1671-3915</td>
  </tr>
  <tr>
    <td>Urban Geology</td>
    <td></td>
    <td>2097-3764</td>
  </tr>
  <tr>
    <td>Engineering and Construction</td>
    <td></td>
    <td>1673-0781</td>
  </tr>
  <tr>
    <td>Engineering and Technological Research</td>
    <td></td>
    <td>2096-2789</td>
  </tr>
  <tr>
    <td>Scientific and Technological Innovation</td>
    <td></td>
    <td>2096-4390</td>
  </tr>
  <tr>
    <td>Engineering & Test</td>
    <td></td>
    <td>1674-3407</td>
  </tr>
  <tr>
    <td>Inner Mongolia Water Resources</td>
    <td></td>
    <td>1009-0408</td>
  </tr>
  <tr>
    <td>China Cement</td>
    <td></td>
    <td>1671-8321</td>
  </tr>
  <tr>
    <td>Guangdong Chemical Industry</td>
    <td></td>
    <td>1007-1865</td>
  </tr>
  <tr>
    <td>Jiangxi Building Materials</td>
    <td></td>
    <td>1006-2890</td>
  </tr>
  <tr>
    <td>Tianjin Mining Technology</td>
    <td></td>
    <td>1004-0945</td>
  </tr>
  <tr>
    <td>Journal of Zhejiang University of Water Resources and Electric Power</td>
    <td></td>
    <td>2095-7092</td>
  </tr>
  <tr>
    <td>China Municipal Engineering</td>
    <td></td>
    <td>1004-4655</td>
  </tr>
  <tr>
    <td>China Storage & Transport</td>
    <td></td>
    <td>1005-0434</td>
  </tr>
</table>

<table>
  <tr>
    <th>Journal Name</th>
    <th>Mining</th>
    <th>ISSN</th>
  </tr>
  <tr>
    <td>Coal Engineering</td>
    <td></td>
    <td>1671-0959</td>
  </tr>
  <tr>
    <td>Mining Engineering</td>
    <td></td>
    <td>1671-8550</td>
  </tr>
  <tr>
    <td>Mechanical Management and Development</td>
    <td></td>
    <td>1003-5773X</td>
  </tr>
  <tr>
    <td>Coal and Chemical Industry</td>
    <td></td>
    <td>2095-5979</td>
  </tr>
  <tr>
    <td>Colliery Mechanical & Electrical Technology</td>
    <td></td>
    <td>1001-0874</td>
  </tr>
  <tr>
    <td>Modern Mining</td>
    <td></td>
    <td>1674-6082</td>
  </tr>
  <tr>
    <td>China Mining Engineering</td>
    <td></td>
    <td>1672-9699X</td>
  </tr>
  <tr>
    <td>Shaanxi Coal Science and Technology</td>
    <td></td>
    <td>1005-2801</td>
  </tr>
  <tr>
    <td>Jiangxi Coal Science & Technology</td>
    <td></td>
    <td>1006-2572</td>
  </tr>
  <tr>
    <td>Metal Mine</td>
    <td></td>
    <td>1001-1250</td>
  </tr>
  <tr>
    <td>Modern Chemical Research</td>
    <td></td>
    <td>1672-9611A</td>
  </tr>
  <tr>
    <td>Petroleum Geology and Engineering</td>
    <td></td>
    <td>1673-8217</td>
  </tr>
  <tr>
    <td>Coal Mine Modernization</td>
    <td></td>
    <td>1009-0797</td>
  </tr>
  <tr>
    <td>Shaanxi Coal</td>
    <td></td>
    <td>1671-749X</td>
  </tr>
  <tr>
    <td>Drilling Engineering</td>
    <td></td>
    <td>2096-9866</td>
  </tr>
  <tr>
    <td>Mineral Resources and Geology</td>
    <td></td>
    <td>1001-5663</td>
  </tr>
  <tr>
    <td>Mine Surveying</td>
    <td></td>
    <td>1001-358X</td>
  </tr>
  <tr>
    <td>Coke</td>
    <td></td>
    <td>1005-2798</td>
  </tr>
  <tr>
    <td>Mining Equipment</td>
    <td></td>
    <td>2095-1418</td>
  </tr>
  <tr>
    <td>Inner Mongolia Coal Economy</td>
    <td></td>
    <td>1008-0155</td>
  </tr>
  <tr>
    <td>Inner Mongolia Petrochemical Industry</td>
    <td></td>
    <td>1006-7981</td>
  </tr>
  <tr>
    <td>Bronze and Metal Conservation</td>
    <td></td>
    <td>2095-8082</td>
  </tr>
  <tr>
    <td>China Plant Engineering</td>
    <td></td>
    <td>1671-0711</td>
  </tr>
  <tr>
    <td>Engineering and Construction</td>
    <td></td>
    <td>1673-5781</td>
  </tr>
  <tr>
    <td>Scientific and Technological Innovation</td>
    <td></td>
    <td>2096-4390</td>
  </tr>
  <tr>
    <td>Engineering & Test</td>
    <td></td>
    <td>1674-3407</td>
  </tr>
  <tr>
    <td>Energy Technology and Management</td>
    <td></td>
    <td>1672-9943</td>
  </tr>
  <tr>
    <td>Coal Technology</td>
    <td></td>
    <td>1008-8725</td>
  </tr>
</table>

Table 4: List of the engineering journals used for construction the benchmark. The information for environment domain and mining domain is shown above, and information for other domains is in Table 9 and 10.

B Template for Extraction

In order to obtain the necessary content for evaluating and judging systems from engineering reports, we manually format a template. When extracting, we combine the report with this template, input it into GPT-4o, and then organize the output into JSON format and save it. The extracted content includes: real-world complex requirements, expert-authored solutions, analytical knowledge used to interpret the requirements, technical knowledge applied in addressing the requirements, and explanations for the expert’s solution design process. The complete template is shown in Figure 6.

C Prompt for Node Expansion

In the growth of the tree, there are two expansion processes: design and review. The review process is divided into two stages: generating proposals based on parent node information and generating comments based on retrieved documents. The design process is also divided into two stages: generating proposals based on parent node information and generating solutions based on retrieved documents. Moreover, the design process based on the root node and the design process based on the comment node use different prompts due to the differences in input information. All the prompts mentioned in this section are shown in Figure 7.

D Prompt for Scores Calculation

In order to evaluate the solutions provided by the system, we follow the methods from previous Long-form QA evaluation (Tan et al., 2024; Wang et al., 2024a; Li et al., 2025b), and use a LLM-based scoring method. Specifically, for a given solution generated by the system, we calculate two scores: (1) Analytical score, which uses the golden solution, explanation, and corresponding analytical knowledge as references, allowing GPT-4o to assess whether the system’s solution sufficiently consider the challenges posed by the complex constraints in the requirements. (2) Technical score,
which uses the golden solution, explanation, and corresponding technical knowledge as references, allowing GPT-4o to evaluate whether the system’s solution correctly apply the appropriate technologies to address the complex constraints in the requirements. Both analytical score and technical score are range from 0 to 100. The used prompts for score calculation are shown in Figure 8.

E   Experiment Results on More LLMs

In order to further demonstrate the effectiveness of SolutionRAG, we conduct experiments using various LLMs as base models. The experimental results show that SolutionRAG still achieves the best performance, proving its strong generalization capability across different base models. Detailed experimental results are shown in Table 5, 6, 7, 8.
The above is a solution for a complex task. You need to extract the following content from the input document, with the following requirements:
1. Extraction must strictly follow the template.
2. Extraction must strictly be based on the document content; no additional information may be added.
3. The extracted content must be detailed, including specific values and professional terms.

{
    "title": "", // The title of the document
    "requirement": "", // A detailed and comprehensive description of the task in the document, covering all the conditions mentioned in the above analysis
    "solution": "This paper proposes the solution of..., specifically, first, considering the conditions of..., and the challenges faced using... technologies, through... achieved... secondly..., # Based on the explanation, a detailed and comprehensive introduction to the solution in the document
    "explanation": [
        {
            "idk": "analysis1",
            "condition": "Condition is...", // The description of task conditions in the document
            "challenge": "Under this condition, conducting... will face challenges such as..., which may lead to... consequences" // Based on the document's analysis, the challenges and potential negative consequences encountered in the task due to this condition
        },
        {
            "technical knowledge": [ // Extract the solutions for the task from the document, with specific values and professional terms
                {
                    "idk": "technology0",
                    "name": "Technology", // A key technology in the solution in the document
                    "detail": "This technology applies to..., and can achieve..." // The scope and effects of the technology mentioned in the document, with specific values and professional terms
                }
            ],
            "explanation": [ // Find the reasoning from the task to the solution in the document, corresponding to the analysis. For each analysis, an explanation must be provided; one analysis may correspond to one or more technologies
                {
                    "idk": "explanation0",
                    "content": "In response to analysis1, considering the conditions of... and the challenges that may arise, the use of technology0... achieves..." // Find the explanation in the document that describes how the mentioned technologies solve the challenges from the analysis
                }
            ]
        }
    ]
}

Example:

{
    "title": "Analysis of Difficulties in High-Temperature and Ultra-Deep Geothermal Well Drilling and Construction Plan Design in Datong Area",
    "requirement": "Under the conditions of complex strata, high-temperature and high-pressure environments, and ultra-deep high geostress, complete the task of drilling a geothermal exploration well to a depth of 4000 meters, requiring full-core sampling, a core recovery rate of 70%, and meeting well inclination and logging quality requirements.",
    "solution": "This paper proposes the solution of..., specifically, first, considering the complexity of the strata, which may lead to hole collapse and stuck pipe, the use of high-temperature drilling fluid technology and wireline coring technologies, through optimizing tools and mud performance, achieved efficient drilling and core recovery; secondly, to address the high-temperature and high-pressure environment, the use of high-temperature-resistant drilling fluid and cementing technology, through high-temperature-resistant formulations and blowout prevention configurations, ensured construction safety and continuity; finally, to address the wall instability issues caused by ultra-deep high geostress, the automatic drilling system and cementing technology were used, ensuring wall stability and progress through stable drilling pressure and high-performance cementing."
    "explanation": [
        {
            "idk": "analysis1",
            "condition": "The condition is complex strata, including Quaternary and Neogene loose strata, and deep rocks of the Precambrian lining group, possibly with fractured rock layers.",
            "challenge": "Under this condition, drilling will face challenges such as hole collapse, excessive wear on drilling tools, low rock drillability, and stuck pipe, which may lead to low construction efficiency, tool damage, or hole instability."
        },
        {
            "technical knowledge": [
                {
                    "idk": "technology0",
                    "name": "High-Temperature Drilling Fluid Technology",
                    "detail": "Technology applies to high-temperature strata, using high-temperature-resistant water-based drilling fluids with a resistance of 240°C and 220°C sealing agents, which can operate stably at 236°C, ensuring hole wall stability and drilling fluid performance."
                },
                {
                    "idk": "technology1",
                    "name": "Wireline Coring Drilling Technology",
                    "detail": "This technology is suitable for deep rock drilling, achieving stable and efficient core sampling in hard rocks by reducing tripping time and improving drilling efficiency."
                },
                {
                    "idk": "technology2",
                    "name": "Automatic Drilling System",
                    "detail": "Suitable for deep hole construction, it modifies traditional drilling rigs to achieve stepless speed regulation and stable control of drilling pressure, meeting the requirement of 1.2 meters/hour drilling speed for deep wells."
                },
                {
                    "idk": "technology3",
                    "name": "Well Control and Cementing Technology",
                    "detail": "Uses 244.5 mm casing and a high-temperature-resistant cement system, with a blowout preventer that has a sealing pressure of 14 MPa, effectively preventing blowout accidents in high-temperature and high-pressure wells."
                }
            ],
            "explanation": [
                {
                    "idk": "explanation0",
                    "content": "In response to analysis1, considering that complex strata may lead to hole collapse and stuck pipe, the use of technology1 and technology3, through efficient tool design and optimized mud performance, achieved stable drilling and high core recovery rates."
                },
                {
                    "idk": "explanation1",
                    "content": "In response to analysis1, considering that high-temperature and high-pressure conditions may trigger blowouts and drilling fluid failure, the use of technology0 and technology3, through high-temperature-resistant formulations and blowout preventer configurations, ensured construction safety and continuity."
                },
                {
                    "idk": "explanation2",
                    "content": "In response to analysis2, considering that ultra-deep high geostress may lead to wall collapse and necking, the use of technology2 and technology3, through stable drilling pressure and high-performance cementing, resolved the complex issues caused by geostress."
                }
            ]
        }
    ]
}

Figure 6: Template used to extract useful content from original engineering reports, aiming to capture real-world complex requirements, expert-authored solutions, analytical knowledge used to interpret the requirements, technical knowledge applied in addressing the requirements, and explanations for the expert’s solution design process.
prompt_for_get_solution_proposal_from_root = """<Instruction>:
In order to solve the following question, I need to search for relevant knowledge in external knowledge bases. Please generate a proposal based on the question and tell me what areas of knowledge I should search for.
<Question>:
[requirement]
<Proposal>:
***

prompt_for_get_comment_proposal = """<Instruction>:
For the following question, there is currently a candidate solution. To evaluate this candidate solution, I need to search for relevant knowledge in external knowledge bases. Please generate a proposal based on the question and the candidate solution, and tell me what areas of knowledge I should search for.
<Question>:
[requirement]
<Candidate Solution>:
[solution]
<Proposal>:***

prompt_for_get_solution_proposal = """<Instruction>:
For the following question, there is a candidate solution as well as an expert's evaluation of this candidate solution. In order to redesign a better solution, I need to search for relevant knowledge in external knowledge bases. Please generate a proposal based on the question and the candidate solution, and tell me what areas of knowledge I should search for.
<Question>:
[requirement]
<Candidate Solution>:
[old_solution]
<Critique for Candidate Solution>:
[comment]
<Proposal>:***

prompt_for_get_solution_for_root = """<Instruction>:
Based on the reference knowledge, design a good solution for the question. Be sure to make full use of reference knowledge to analyze the challenges contained within the question and provide a comprehensive solution.
<Question>:
[question]
<Reference>:
[reference]
<Solution>:***

prompt_for_get_doubt = """<Instruction>:
For the following question, a candidate solution has already been provided. You need to critique the candidate solution based on the reference knowledge. Be sure to make full use of the reference knowledge to identify the shortcomings of the old solution in terms of its analysis of the challenges in the question and its technical implementation.
<Question>:
[question]
<Candidate Solution>:
[solution]
<Reference>:
[reference]
<Critique>:***

prompt_for_get_solution = """<Instruction>:
For the following question, an old solution has already been provided and its shortcomings have been pointed out by human experts. You need to redesign a better solution based on the reference knowledge and the guidance from human experts. Be sure to make full use of the reference knowledge to analyze the challenges contained within the question and provide a comprehensive solution.
<Question>:
[question]
<Candidate Solution>:
[old_solution]
<Critique for Candidate Solution>:
[comment]
<Reference>:
[reference]
<New Solution>:***

Figure 7: Prompts used in node expansion of tree growth, including generating solution proposals and solutions based on the root node, generating comment proposals and comments based on a solution node, and generating solution proposals and solutions based on a comment node.
<<Task>>
[task]
<<Model-generated solution>>
[solution]
<<Judgement reference>>
# Analysis knowledge:
[analysis knowledge]
# Technology knowledge:
[technology knowledge]
# Golden explanation:
[golden explanation]
# Golden solution:
[golden solution]
<<Instruction>>
The above <<Task>> is a complex requirement in an actual engineering scenario. The above <<Model-generated solution>> is a solution generated by a certain model. You are required to evaluate this solution based on the <<Judgement reference>> annotated by a human expert. The <<Judgement reference>> consists of the following four parts:
(a) Analysis knowledge: A deep analysis of the various restrictive factors present in this complex requirement.
(b) Technology knowledge: A detailed explanation of the various technologies that must be used to solve this complex requirement.
(c) Golden explanation: The golden explanation provided by human experts to overcome various challenges.
(d) Golden solution: The standard solution provided by human experts.
Your evaluation of the <<Model-generated solution>> must fully consider the <<Judgement reference>>. The specific evaluation requirements are as follows.

<<Requirements>>
1. Please evaluate the solution above from 2 dimensions. The range for each of the three scores is an integer between 0 and 100, where the minimum score is 0 and the maximum score is 100.
2. The scoring details for the 2 dimensions are as follows:
(1) Analysis knowledge: Refer to the aforementioned Analysis knowledge, Golden explanation, and Golden solution to assess whether the <<Model-generated solution>> has thoroughly considered the various restrictive factors in the <<Task>>. Pay special attention to listing each restrictive factor in the Analysis knowledge one by one and whether this part is sufficiently correct and specific.
    (2.1) If no factors are considered, score 0.
    (2.2) If factors are considered but the analysis is not entirely correct, score 11-30 depending on the degree of correctness.
    (2.3) If factors are considered and the analysis is correct but not specific, score 31-60 depending on the level of specificity.
    (2.4) If factors are considered and the analysis is correct and specific, score 61-90 based on its similarity to the standard Analysis knowledge.
    (2.5) If it is fully consistent with the standard Analysis knowledge, score 100.
(2) Technology Score: Refer to the aforementioned Technology knowledge, Golden explanation, and Golden solution to evaluate whether the <<Model-generated solution>> utilizes the technology, and whether this part is sufficiently correct and specific.
    (2.1) If technologies are used but not entirely correctly, score 11-30 depending on the degree of correctness.
    (2.2) If technologies are used correctly but not specifically, score 31-60 depending on the level of specificity.
    (2.3) If technologies are used correctly and specifically, score 61-90 based on their similarity to the standard Technology knowledge.
    (2.4) If it is fully consistent with the standard Technology knowledge, score 100.
3. For each dimension, please output the identifier ##Score## followed by your evaluation results in the form of a dictionary, i.e., ##Scores## {"Analysis Score": int, "Technology Score": int}
4. Note that a longer solution is not necessarily better. The sole basis of your evaluation process is the aforementioned Judgement reference, and your evaluation results must fully take this reference into account.
5. Please refer to the example below when completing your evaluation:
##Example of evaluation:
For Analysis Score Evaluation: The analysis evaluates whether the solution adequately considers restrictive factors in the task. Let's analyze each factor from the Analysis knowledge:
For (analysis 1) Uneven soil particle distribution and high groundwater levels: The solution mentions groundwater analysis (in Geotechnical Investigation and Analysis) and discusses the use of casing interlocking pile technology, which can effectively control groundwater intrusion. However, the solution does not mention using sheet piles to stabilize the foundation pit. While the factors are considered, the analysis lacks detail and specificity regarding solutions such as technology for groundwater cut-off and particle stabilization. Conclusion: Analysis is present and mostly correct but lacks specificity in addressing key risks like piping and groundwater intrusion. Score: 60/100.
For (analysis 2) Sensitive surrounding environment: The solution mentions sensitivity in the surrounding environment and mentions real-time monitoring and mitigation measures for underground pipelines or nearby structures, such as dynamic monitoring and quantified safety parameters. Conclusion: The analysis is correct but not specific enough regarding risks to surrounding buildings and underground pipelines. Score: 60/100.
For (analysis 3) Dynamic Monitoring and Protection Measures: The solution does not address all of them comprehensively or with sufficient specificity. Final Analysis Score: 57/100.
For Technology Score Evaluation: The solution checks whether appropriate technologies were employed. Let's analyze each technology from the Technology knowledge:
For (technology 1) Casing Interlocking Pile Technology: The solution does not explicitly mention casing interlocking pile technology, which is critical for ensuring foundation pit enclosure and managing groundwater effectively. Instead, it suggests general retaining structures like sheet piles and diaphragm walls, which are less specific and not as effective. Conclusion: Partially applied. Score: 50/100.
For (technology 2) Precipitation and Grouting Measures: The solution mentions phased excavation and the use of steel supports and bracing but does not include details about pre-applied axial forces, which are vital for controlling deformation and ensuring stability. The specificity of this technology's application is lacking. Conclusion: Partially applied. Score: 50/100.
For (technology 3) Dynamic Monitoring and Protection Measures: The solution discusses dewatering systems and drainage channels but omits the use of grouting measures, which are essential for controlling groundwater levels and preventing leakage. Grouting was explicitly required in the reference but is absent here. Conclusion: Partially applied. Score: 40/100.
For (technology 4) Dynamic Monitoring and Protection Measures: The solution includes real-time monitoring with sensors and references emergency response measures. However, it does not specify how to monitor settlement rates, or specify dynamic adjustment strategies. These omissions limit the specificity of this technology's application. Conclusion: Partially applied. Score: 50/100.
Final Technology Score: The solution employs some relevant technologies but omits critical ones (e.g., casing interlocking piles and grouting measures) and lacks specificity in others. Final Technology Score = 57/100.
Final Scores
##Scores## [{"Analysis Score": 57, "Technology Score": 35}]

<<Judgement>>
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Env.</th>
    <th colspan="2">Min.</th>
    <th colspan="2">Tra.</th>
    <th colspan="2">Aer.</th>
    <th colspan="2">Tel.</th>
    <th colspan="2">Arc.</th>
    <th colspan="2">Wat.</th>
    <th colspan="2">Far.</th>
  </tr>
  <tr>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
  </tr>
  <tr>
    <td colspan="18">Single-round RAG Methods</td>
  </tr>
  <tr>
    <td>Naïve-RAG (Lewis et al., 2020)</td><td>59.7</td><td>56.9</td><td>55.6</td><td>37.8</td><td>60.8</td><td>56.8</td><td>68.6</td><td>66.7</td><td>69.4</td><td>67.5</td><td>68.4</td><td>63.5</td><td>64.6</td><td>56.8</td><td>62.6</td><td>59.2</td>
  </tr>
  <tr>
    <td>Rerank-RAG (Li et al., 2023)</td><td>61.2</td><td>58.0</td><td>54.2</td><td>37.9</td><td>60.6</td><td>56.2</td><td>68.5</td><td>65.0</td><td>66.4</td><td>67.8</td><td>63.1</td><td>59.2</td><td>62.2</td><td>62.7</td><td>68.2</td><td>56.6</td>
  </tr>
  <tr>
    <td colspan="18">Multi-round RAG Methods</td>
  </tr>
  <tr>
    <td>Self-RAG (Asai et al., 2024)</td><td>60.3</td><td>62.0</td><td>56.8</td><td>40.3</td><td>61.1</td><td>56.5</td><td>65.4</td><td>64.2</td><td>70.5</td><td>71.4</td><td>69.2</td><td>64.8</td><td>60.8</td><td>59.5</td><td>66.6</td><td>64.4</td>
  </tr>
  <tr>
    <td>GenGround (Shi et al., 2024)</td><td>43.8</td><td>36.5</td><td>38.2</td><td>21.5</td><td>52.6</td><td>41.8</td><td>53.5</td><td>39.6</td><td>62.5</td><td>48.3</td><td>53.8</td><td>45.8</td><td>54.9</td><td>47.2</td><td>55.8</td><td>42.7</td>
  </tr>
  <tr>
    <td>RQ-RAG (Chan et al., 2024)</td><td>53.3</td><td>43.8</td><td>45.4</td><td>21.7</td><td>54.0</td><td>45.0</td><td>53.3</td><td>43.5</td><td>59.3</td><td>47.6</td><td>56.2</td><td>48.1</td><td>57.8</td><td>45.6</td><td>62.2</td><td>50.6</td>
  </tr>
  <tr>
    <td colspan="18">Tree-based Exploration and Bi-point Thinking</td>
  </tr>
  <tr>
    <td>SolutionRAG (Ours)</td><td>63.0</td><td>63.1</td><td>58.0</td><td>42.4</td><td>64.2</td><td>62.4</td><td>70.8</td><td>68.2</td><td>72.2</td><td>73.0</td><td>71.0</td><td>71.3</td><td>65.9</td><td>65.2</td><td>70.2</td><td>65.6</td>
  </tr>
</table>

Table 5: Main experimental results based on Mistral-7B-Instruct-v0.3.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Env.</th>
    <th colspan="2">Min.</th>
    <th colspan="2">Tra.</th>
    <th colspan="2">Aer.</th>
    <th colspan="2">Tel.</th>
    <th colspan="2">Arc.</th>
    <th colspan="2">Wat.</th>
    <th colspan="2">Far.</th>
  </tr>
  <tr>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
  </tr>
  <tr>
    <td colspan="18">Single-round RAG Methods</td>
  </tr>
  <tr>
    <td>Naïve-RAG (Lewis et al., 2020)</td><td>54.3</td><td>57.0</td><td>51.2</td><td>41.1</td><td>56.4</td><td>52.1</td><td>65.9</td><td>68.1</td><td>65.2</td><td>59.8</td><td>61.6</td><td>59.9</td><td>61.3</td><td>54.4</td><td>66.2</td><td>65.0</td>
  </tr>
  <tr>
    <td>Rerank-RAG (Li et al., 2023)</td><td>51.7</td><td>52.3</td><td>48.4</td><td>31.9</td><td>54.6</td><td>51.6</td><td>62.3</td><td>64.8</td><td>63.3</td><td>63.4</td><td>59.8</td><td>53.1</td><td>60.9</td><td>49.6</td><td>65.9</td><td>59.4</td>
  </tr>
  <tr>
    <td colspan="18">Multi-round RAG Methods</td>
  </tr>
  <tr>
    <td>Self-RAG (Asai et al., 2024)</td><td>55.8</td><td>56.5</td><td>49.2</td><td>32.2</td><td>57.0</td><td>51.5</td><td>64.0</td><td>66.7</td><td>63.6</td><td>63.3</td><td>65.1</td><td>60.8</td><td>59.9</td><td>54.5</td><td>65.8</td><td>65.9</td>
  </tr>
  <tr>
    <td>GenGround (Shi et al., 2024)</td><td>48.9</td><td>43.0</td><td>33.2</td><td>26.9</td><td>46.7</td><td>38.2</td><td>51.6</td><td>45.3</td><td>46.8</td><td>34.6</td><td>49.0</td><td>37.2</td><td>50.1</td><td>39.4</td><td>51.0</td><td>42.8</td>
  </tr>
  <tr>
    <td>RQ-RAG (Chan et al., 2024)</td><td>46.1</td><td>39.7</td><td>41.8</td><td>17.9</td><td>51.1</td><td>40.4</td><td>47.7</td><td>33.8</td><td>51.6</td><td>38.0</td><td>54.3</td><td>43.3</td><td>48.2</td><td>36.2</td><td>55.4</td><td>46.6</td>
  </tr>
  <tr>
    <td colspan="18">Tree-based Exploration and Bi-point Thinking</td>
  </tr>
  <tr>
    <td>SolutionRAG (Ours)</td><td>56.9</td><td>57.6</td><td>53.9</td><td>42.0</td><td>58.6</td><td>53.0</td><td>68.8</td><td>70.2</td><td>65.6</td><td>65.1</td><td>66.2</td><td>61.8</td><td>62.6</td><td>56.8</td><td>67.3</td><td>68.8</td>
  </tr>
</table>

Table 6: Main experimental results based on Llama-3.1-8B-Instruct.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Env.</th>
    <th colspan="2">Min.</th>
    <th colspan="2">Tra.</th>
    <th colspan="2">Aer.</th>
    <th colspan="2">Tel.</th>
    <th colspan="2">Arc.</th>
    <th colspan="2">Wat.</th>
    <th colspan="2">Far.</th>
  </tr>
  <tr>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
  </tr>
  <tr>
    <td colspan="18">Single-round RAG Methods</td>
  </tr>
  <tr>
    <td>Naïve-RAG (Lewis et al., 2020)</td><td>61.2</td><td>58.0</td><td>56.4</td><td>36.0</td><td>62.4</td><td>54.7</td><td>67.0</td><td>64.2</td><td>62.6</td><td>65.6</td><td>66.2</td><td>62.1</td><td>63.6</td><td>54.0</td><td>64.4</td><td>61.5</td>
  </tr>
  <tr>
    <td>Rerank-RAG (Li et al., 2023)</td><td>62.1</td><td>60.2</td><td>52.8</td><td>34.4</td><td>59.2</td><td>48.3</td><td>64.2</td><td>65.1</td><td>63.4</td><td>61.3</td><td>62.2</td><td>57.0</td><td>64.7</td><td>53.7</td><td>57.9</td><td>59.6</td>
  </tr>
  <tr>
    <td colspan="18">Multi-round RAG Methods</td>
  </tr>
  <tr>
    <td>Self-RAG (Asai et al., 2024)</td><td>63.0</td><td>64.2</td><td>54.1</td><td>35.0</td><td>62.9</td><td>55.4</td><td>68.6</td><td>69.2</td><td>64.5</td><td>65.4</td><td>64.5</td><td>64.7</td><td>63.9</td><td>56.6</td><td>61.0</td><td>58.6</td>
  </tr>
  <tr>
    <td>GenGround (Shi et al., 2024)</td><td>50.1</td><td>37.5</td><td>50.6</td><td>29.9</td><td>53.3</td><td>36.0</td><td>54.8</td><td>45.3</td><td>56.5</td><td>48.3</td><td>57.8</td><td>50.2</td><td>58.2</td><td>46.5</td><td>58.1</td><td>51.4</td>
  </tr>
  <tr>
    <td>RQ-RAG (Chan et al., 2024)</td><td>52.7</td><td>43.7</td><td>47.9</td><td>25.5</td><td>51.6</td><td>37.1</td><td>54.1</td><td>44.9</td><td>57.1</td><td>43.5</td><td>54.2</td><td>44.6</td><td>49.9</td><td>38.9</td><td>58.0</td><td>44.3</td>
  </tr>
  <tr>
    <td colspan="18">Tree-based Exploration and Bi-point Thinking</td>
  </tr>
  <tr>
    <td>SolutionRAG (Ours)</td><td>65.6</td><td>64.2</td><td>58.6</td><td>44.6</td><td>63.6</td><td>57.7</td><td>69.8</td><td>71.5</td><td>66.1</td><td>67.8</td><td>67.6</td><td>66.9</td><td>64.9</td><td>59.8</td><td>66.9</td><td>64.3</td>
  </tr>
</table>

Table 7: Main experimental results based on Qwen2.5-3B-Instruct.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Env.</th>
    <th colspan="2">Min.</th>
    <th colspan="2">Tra.</th>
    <th colspan="2">Aer.</th>
    <th colspan="2">Tel.</th>
    <th colspan="2">Arc.</th>
    <th colspan="2">Wat.</th>
    <th colspan="2">Far.</th>
  </tr>
  <tr>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
    <th>AS</th><th>TS</th>
  </tr>
  <tr>
    <td colspan="18">Single-round RAG Methods</td>
  </tr>
  <tr>
    <td>Naïve-RAG (Lewis et al., 2020)</td><td>49.0</td><td>41.5</td><td>47.0</td><td>25.3</td><td>54.7</td><td>46.6</td><td>55.5</td><td>52.7</td><td>58.4</td><td>55.9</td><td>56.2</td><td>50.8</td><td>53.5</td><td>43.5</td><td>61.3</td><td>51.6</td>
  </tr>
  <tr>
    <td>Rerank-RAG (Li et al., 2023)</td><td>50.6</td><td>43.5</td><td>50.2</td><td>31.2</td><td>50.0</td><td>41.2</td><td>53.9</td><td>49.9</td><td>59.5</td><td>54.8</td><td>50.6</td><td>40.8</td><td>54.6</td><td>49.1</td><td>60.6</td><td>52.8</td>
  </tr>
  <tr>
    <td colspan="18">Multi-round RAG Methods</td>
  </tr>
  <tr>
    <td>Self-RAG (Asai et al., 2024)</td><td>49.6</td><td>44.2</td><td>48.3</td><td>30.4</td><td>51.5</td><td>45.6</td><td>53.7</td><td>47.5</td><td>57.7</td><td>53.8</td><td>56.4</td><td>51.9</td><td>55.7</td><td>48.8</td><td>57.8</td><td>53.0</td>
  </tr>
  <tr>
    <td>GenGround (Shi et al., 2024)</td><td>23.4</td><td>12.4</td><td>22.5</td><td>16.0</td><td>34.7</td><td>32.2</td><td>32.1</td><td>22.6</td><td>41.1</td><td>36.0</td><td>26.5</td><td>19.5</td><td>35.0</td><td>30.6</td><td>42.6</td><td>43.8</td>
  </tr>
  <tr>
    <td>RQ-RAG (Chan et al., 2024)</td><td>42.8</td><td>29.6</td><td>46.0</td><td>25.0</td><td>46.5</td><td>33.1</td><td>41.6</td><td>34.6</td><td>45.4</td><td>30.9</td><td>46.3</td><td>35.4</td><td>50.1</td><td>38.2</td><td>49.3</td><td>37.7</td>
  </tr>
  <tr>
    <td colspan="18">Tree-based Exploration and Bi-point Thinking</td>
  </tr>
  <tr>
    <td>SolutionRAG (Ours)</td><td>51.6</td><td>46.2</td><td>50.6</td><td>33.2</td><td>58.1</td><td>49.6</td><td>57.0</td><td>56.7</td><td>61.5</td><td>57.7</td><td>58.1</td><td>53.3</td><td>56.2</td><td>50.4</td><td>62.1</td><td>54.2</td>
  </tr>
</table>

Table 8: Main experimental results based on Qwen2.5-0.5B-Instruct.
<table>
  <tr>
    <th>Journal Name</th>
    <th>ISSN</th>
  </tr>
  <tr>
    <td>Railway Construction Technology</td>
    <td>1009-4539</td>
  </tr>
  <tr>
    <td>Northern Communications</td>
    <td>1673-6052</td>
  </tr>
  <tr>
    <td>China Municipal Engineering</td>
    <td>1004-4655</td>
  </tr>
  <tr>
    <td>Highway</td>
    <td>0451-0712</td>
  </tr>
  <tr>
    <td>Rural Roads Bridges & Flood Control</td>
    <td>1009-7872</td>
  </tr>
  <tr>
    <td>Technology Innovation & Application</td>
    <td>2005-2945</td>
  </tr>
  <tr>
    <td>Marine Equipment/Materials & Marketing</td>
    <td>1006-6969</td>
  </tr>
  <tr>
    <td>Engineering and Construction</td>
    <td>1673-5781</td>
  </tr>
  <tr>
    <td>Port Operation</td>
    <td>1000-8999</td>
  </tr>
  <tr>
    <td>Structure Engineers</td>
    <td>1005-0159</td>
  </tr>
  <tr>
    <td>China Highway</td>
    <td>1006-8997</td>
  </tr>
  <tr>
    <td>Engineering and Technological Research</td>
    <td>2096-2789</td>
  </tr>
  <tr>
    <td>Construction Machinery Technology & Management</td>
    <td>1004-0005</td>
  </tr>
  <tr>
    <td>TranspoWorld</td>
    <td>1006-8872</td>
  </tr>
  <tr>
    <td>Railway Navigation and Surveying</td>
    <td>1672-7747</td>
  </tr>
  <tr>
    <td>Transport Construction & Management</td>
    <td>1673-8098</td>
  </tr>
  <tr>
    <td>Guangdong Water Resources and Hydropower</td>
    <td>1008-0112</td>
  </tr>
  <tr>
    <td>Western China Communications Science & Technology</td>
    <td>1673-4874</td>
  </tr>
  <tr>
    <td>Jiangsu Science and Technology Information</td>
    <td>1004-7530</td>
  </tr>
  <tr>
    <td>Shanghai Engineering</td>
    <td>1006-8464</td>
  </tr>
  <tr>
    <td>Hoisting and Conveying Machinery</td>
    <td>1001-0785</td>
  </tr>
  <tr>
    <td>Jiangxi Building Materials</td>
    <td>1006-2890</td>
  </tr>
  <tr>
    <td>Scientific and Technological Innovation</td>
    <td>2096-4390</td>
  </tr>
  <tr>
    <td>Transport Business China</td>
    <td>1673-3681</td>
  </tr>
  <tr>
    <td>Shacman Cement</td>
    <td>0451-0712</td>
  </tr>
</table>

<table>
  <tr>
    <th>Journal Name</th>
    <th>ISSN</th>
  </tr>
  <tr>
    <td>Spacecraft Engineering</td>
    <td>1673-8748</td>
  </tr>
  <tr>
    <td>Aeronautical Manufacturing Technology</td>
    <td>1671-833X</td>
  </tr>
  <tr>
    <td>Aviation Maintenance & Engineering</td>
    <td>1672-0989</td>
  </tr>
  <tr>
    <td>Journal of Ordnance Equipment Engineering</td>
    <td>2096-2304</td>
  </tr>
  <tr>
    <td>Aeroginone</td>
    <td>2096-2304</td>
  </tr>
  <tr>
    <td>International</td>
    <td>2096-2304</td>
  </tr>
  <tr>
    <td>Avionics Technology</td>
    <td>1006-141X</td>
  </tr>
  <tr>
    <td>System Simulation Technology</td>
    <td>1673-1964</td>
  </tr>
  <tr>
    <td>Journal of Civil Aviation</td>
    <td>2096-4994</td>
  </tr>
  <tr>
    <td>Safety & EMC</td>
    <td>1005-6547</td>
  </tr>
  <tr>
    <td>Internal Combustion Engine & Parts</td>
    <td>1674-957X</td>
  </tr>
  <tr>
    <td>Aeronautical Computing Technique</td>
    <td>1671-654X</td>
  </tr>
  <tr>
    <td>Meteorological Science and Technology</td>
    <td>1671-6345</td>
  </tr>
  <tr>
    <td>Journal of Astronautics</td>
    <td>1000-1328</td>
  </tr>
  <tr>
    <td>Communications Technology</td>
    <td>1002-0492</td>
  </tr>
  <tr>
    <td>China Optoelectronics Progress</td>
    <td>1006-4125</td>
  </tr>
  <tr>
    <td>Engineering & Test</td>
    <td>1674-3407</td>
  </tr>
  <tr>
    <td>Chinese Space Science and Technology</td>
    <td>1000-758X</td>
  </tr>
  <tr>
    <td>Ship Electronic Engineering</td>
    <td>1672-9730</td>
  </tr>
  <tr>
    <td>China Science and Technology Information</td>
    <td>1672-9730</td>
  </tr>
  <tr>
    <td>Deep Sea Exploration</td>
    <td>2096-9287</td>
  </tr>
  <tr>
    <td>China Educational Technology & Equipment</td>
    <td>1671-489X</td>
  </tr>
  <tr>
    <td>Micromotors</td>
    <td>1671-489X</td>
  </tr>
  <tr>
    <td>Spacecraft Recovery & Remote Sensing</td>
    <td>1009-8518</td>
  </tr>
  <tr>
    <td>Journal of Chengdu Aeronautic Polytechnic</td>
    <td>1671-4024</td>
  </tr>
</table>

<table>
  <tr>
    <th>Journal Name</th>
    <th>ISSN</th>
  </tr>
  <tr>
    <td>Systems Engineering and Electronics</td>
    <td>1001-506X</td>
  </tr>
  <tr>
    <td>Electronic Technology & Software Engineering</td>
    <td>2095-5650</td>
  </tr>
  <tr>
    <td>Video Engineering</td>
    <td>1002-8692</td>
  </tr>
  <tr>
    <td>Telecom Engineering Techniques and Standardization</td>
    <td>1008-5599</td>
  </tr>
  <tr>
    <td>Radio & Television Network</td>
    <td>2096-8498</td>
  </tr>
  <tr>
    <td>Telecommunications</td>
    <td>1005-8738</td>
  </tr>
  <tr>
    <td>Electronics Quality</td>
    <td>1003-0107</td>
  </tr>
  <tr>
    <td>Radio & Television Information</td>
    <td>1007-1997</td>
  </tr>
  <tr>
    <td>Automation in Petro-Chemical Industry</td>
    <td>1007-5324</td>
  </tr>
  <tr>
    <td>Telecommunications Engineering</td>
    <td>1000-0801</td>
  </tr>
  <tr>
    <td>Computer Knowledge and Technology</td>
    <td>1009-3044</td>
  </tr>
  <tr>
    <td>Journal of Electronics & Information Technology</td>
    <td>1009-5896</td>
  </tr>
  <tr>
    <td>Laser & Optoelectronics Progress</td>
    <td>1006-4125</td>
  </tr>
  <tr>
    <td>China Digital Cable TV</td>
    <td>1007-5322</td>
  </tr>
  <tr>
    <td>Radio & Television</td>
    <td>1003-3106</td>
  </tr>
  <tr>
    <td>Journal of Beijing Electronic Science and Technology Institute</td>
    <td>1672-464X</td>
  </tr>
  <tr>
    <td>Laser Journal</td>
    <td>0253-2743</td>
  </tr>
  <tr>
    <td>Designing Techniques of Posts and Telecommunications</td>
    <td>1007-3041</td>
  </tr>
  <tr>
    <td>Wireless Internet Science and Technology</td>
    <td>1672-464X</td>
  </tr>
  <tr>
    <td>Journal of University of South China(Science and Technology)</td>
    <td>1673-0062</td>
  </tr>
  <tr>
    <td>Audio Engineering</td>
    <td>1002-8684</td>
  </tr>
  <tr>
    <td>Automation Application</td>
    <td>1674-778X</td>
  </tr>
  <tr>
    <td>Chinese Journal of Lasers</td>
    <td>0258-7025</td>
  </tr>
  <tr>
    <td>Journal of Smart Agriculture</td>
    <td>2096-9902</td>
  </tr>
</table>

Table 9: List of the engineering journals used for construction the benchmark.

<table>
  <tr>
    <th>Journal Name</th>
    <th>ISSN</th>
  </tr>
  <tr>
    <td>Building Technology Development</td>
    <td>1001-523X</td>
  </tr>
  <tr>
    <td>Building Structure</td>
    <td>1002-848X</td>
  </tr>
  <tr>
    <td>Construction & Design for Engineering</td>
    <td>1007-9467</td>
  </tr>
  <tr>
    <td>Modern Paint & Finishing</td>
    <td>1007-9548</td>
  </tr>
  <tr>
    <td>Agricultural Mechanization</td>
    <td>1000-1476</td>
  </tr>
  <tr>
    <td>Theoretical Research in Urban Construction</td>
    <td>2095-2104</td>
  </tr>
  <tr>
    <td>Urban Architecture Space</td>
    <td>2097-1141</td>
  </tr>
  <tr>
    <td>Art and Design</td>
    <td>1008-2832</td>
  </tr>
  <tr>
    <td>Architecture & Culture</td>
    <td>1672-4909</td>
  </tr>
  <tr>
    <td>Journal of Yunnan Polytechnic College</td>
    <td>1008-3693</td>
  </tr>
  <tr>
    <td>Heating Ventilating & Air Conditioning</td>
    <td>1002-8501</td>
  </tr>
  <tr>
    <td>Construction Machinery & Maintenance</td>
    <td>1006-2114</td>
  </tr>
  <tr>
    <td>China Science and Technology Information</td>
    <td>1001-8972</td>
  </tr>
  <tr>
    <td>Construction Machinery and Equipment</td>
    <td>1001-1212</td>
  </tr>
  <tr>
    <td>Innovation of Machinery Technology</td>
    <td>1009-4767</td>
  </tr>
  <tr>
    <td>Jiangxi Building Materials</td>
    <td>1006-2890</td>
  </tr>
  <tr>
    <td>Urban Roads Bridges & Flood Control</td>
    <td>1009-7716</td>
  </tr>
  <tr>
    <td>Fujian Construction Science & Technology</td>
    <td>1006-3943</td>
  </tr>
  <tr>
    <td>Sichuan Construction Science & Technology</td>
    <td>1007-6344</td>
  </tr>
  <tr>
    <td>Engineer and Technological Research</td>
    <td>2096-2789</td>
  </tr>
  <tr>
    <td>Journal of North China Institute of Science and Technology</td>
    <td>1672-7169</td>
  </tr>
  <tr>
    <td>Tianjin Construction Science and Technology</td>
    <td>1008-3197</td>
  </tr>
  <tr>
    <td>World Forestry Research</td>
    <td>1001-4241</td>
  </tr>
  <tr>
    <td>Jiangsu Building Materials</td>
    <td>1004-5538</td>
  </tr>
  <tr>
    <td>Shanghai Construction Science & Technology</td>
    <td>1005-6637</td>
  </tr>
</table>

<table>
  <tr>
    <th>Journal Name</th>
    <th>ISSN</th>
  </tr>
  <tr>
    <td>Design of Water Resources & Hydroelectric Engineering</td>
    <td>1007-6980</td>
  </tr>
  <tr>
    <td>Hydro Science and Cold Zone Engineering</td>
    <td>2096-5419</td>
  </tr>
  <tr>
    <td>Journal of Water Resources and Architectural Engineering</td>
    <td>1672-1144</td>
  </tr>
  <tr>
    <td>Mechanical & Electrical Technique of Hydropower Station</td>
    <td>1672-5387</td>
  </tr>
  <tr>
    <td>Yangtze River</td>
    <td>1001-4179</td>
  </tr>
  <tr>
    <td>Port & Waterway Engineering</td>
    <td>1002-4072</td>
  </tr>
  <tr>
    <td>Technical Supervision in Water Resources</td>
    <td>1008-1305</td>
  </tr>
  <tr>
    <td>Small Hydro Power</td>
    <td>1007-7642</td>
  </tr>
  <tr>
    <td>Pearl River</td>
    <td>1001-9235</td>
  </tr>
  <tr>
    <td>Water Conservancy Construction and Management</td>
    <td>2097-9228</td>
  </tr>
  <tr>
    <td>Water Conservancy Science and Technology and Economy</td>
    <td>1006-2175</td>
  </tr>
  <tr>
    <td>Water Resources Planning and Design</td>
    <td>1672-2469</td>
  </tr>
  <tr>
    <td>Construction Quality</td>
    <td>1671-3702</td>
  </tr>
  <tr>
    <td>Henan Water Resources and South-to-North Water Diversion</td>
    <td>1673-8853</td>
  </tr>
  <tr>
    <td>Engineering and Construction</td>
    <td>1673-8791</td>
  </tr>
  <tr>
    <td>Technological Market</td>
    <td>1006-8554</td>
  </tr>
  <tr>
    <td>Beijing Water</td>
    <td>1673-4637</td>
  </tr>
  <tr>
    <td>Port Engineering Technology</td>
    <td>2097-3519</td>
  </tr>
  <tr>
    <td>Water Resources & Hydropower of Northeast China</td>
    <td>1002-0624</td>
  </tr>
  <tr>
    <td>Mechanical and Electrical Information</td>
    <td>1671-0797</td>
  </tr>
  <tr>
    <td>Maritime Science</td>
    <td>2097-1745</td>
  </tr>
  <tr>
    <td>Gansu Water Resources and Hydropower Technology</td>
    <td>2095-0144</td>
  </tr>
  <tr>
    <td>Water Power</td>
    <td>0559-9342</td>
  </tr>
  <tr>
    <td>Shanxi Water Resources</td>
    <td>1004-7042</td>
  </tr>
  <tr>
    <td>Hainan Water Resources</td>
    <td>1004-7328</td>
  </tr>
</table>

<table>
  <tr>
    <th>Journal Name</th>
    <th>ISSN</th>
  </tr>
  <tr>
    <td>Modern Agricultural Science and Technology</td>
    <td>1007-5739</td>
  </tr>
  <tr>
    <td>Farm Machinery</td>
    <td>1000-9868</td>
  </tr>
  <tr>
    <td>Cereal & Feed Industry</td>
    <td>1003-6202</td>
  </tr>
  <tr>
    <td>Journal of Agricultural Mechanization Research</td>
    <td>1003-188X</td>
  </tr>
  <tr>
    <td>Forestry Machinery & Woodworking Equipment</td>
    <td>2095-2953</td>
  </tr>
  <tr>
    <td>Transactions of the Chinese Society of Agricultural Engineering</td>
    <td>1001-1498</td>
  </tr>
  <tr>
    <td>Forest Research</td>
    <td>1001-1498</td>
  </tr>
  <tr>
    <td>Times Agricultural Machinery</td>
    <td>2095-980X</td>
  </tr>
  <tr>
    <td>Protection Forest Science and Technology</td>
    <td>1005-5215</td>
  </tr>
  <tr>
    <td>Journal of Beijing University of Agriculture</td>
    <td>1002-3186</td>
  </tr>
  <tr>
    <td>Contemporary Horticulture</td>
    <td>1006-4988</td>
  </tr>
  <tr>
    <td>China Southern Agricultural Machinery</td>
    <td>1672-3872</td>
  </tr>
  <tr>
    <td>Forest Inventory and Planning</td>
    <td>1671-3168</td>
  </tr>
  <tr>
    <td>Agricultural Machinery Using & Maintenance</td>
    <td>2097-4515</td>
  </tr>
  <tr>
    <td>Journal of Green Science and Technology</td>
    <td>1674-9944</td>
  </tr>
  <tr>
    <td>China Forestry Products Industry</td>
    <td>1001-8299</td>
  </tr>
  <tr>
    <td>Forestry Machinery & Woodworking Equipment</td>
    <td>2095-2953</td>
  </tr>
  <tr>
    <td>The Food Industry</td>
    <td>1004-471X</td>
  </tr>
  <tr>
    <td>Journal of Hebei Forestry Science and Technology</td>
    <td>1002-3356</td>
  </tr>
  <tr>
    <td>Electrical Automation</td>
    <td>1000-3886</td>
  </tr>
  <tr>
    <td>Journal of Library and Information Science</td>
    <td>2096-1162</td>
  </tr>
  <tr>
    <td>Forest Science and Technology</td>
    <td>2097-0285</td>
  </tr>
  <tr>
    <td>Chinese Journal of Ecology</td>
    <td>1000-4890</td>
  </tr>
  <tr>
    <td>Popular Standardization</td>
    <td>1007-1350</td>
  </tr>
  <tr>
    <td>Management & Technology of SME:</td>
    <td>1673-1069</td>
  </tr>
</table>

Table 10: List of the engineering journals used for construction the benchmark.

Reviewer paper summaries: The paper introduces **SolutionBench**, a new benchmark for evaluating AI systems on the task of complex engineering solution design, which requires generating complete, feasible solutions under multiple real-world constraints. 

To address this challenge, the authors propose **SolutionRAG**, a novel Retrieval-Augmented Generation (RAG) system that combines **tree-based exploration** and a **bi-point thinking mechanism**—alternating between solution generation and critique—to iteratively refine engineering plans. SolutionRAG effectively prunes less promising paths using LLM-based node evaluation, balancing reliability and efficiency. 

Experiments across eight engineering domains show that SolutionRAG significantly outperforms existing deep reasoning and RAG methods in analytical and technical accuracy. This work establishes a new direction for AI-driven engineering design, supported by high-quality data and a structured solution refinement framework.
Summary of strengths: 1. **Introduction of a Novel Task and Benchmark (SolutionBench):**  

   The authors identify and address a meaningful gap in current RAG research by introducing the *complex engineering solution design* task—distinct from traditional multi-hop or long-form QA—where outputs must be complete, feasible, and satisfy multiple real-world constraints. 

2. **Innovative Methodology: SolutionRAG System:**  

   The paper proposes **SolutionRAG**, a novel RAG framework that introduces *tree-based exploration* and *bi-point thinking*. Unlike traditional chain-of-thought or iterative RAG approaches, SolutionRAG structures inference as a **bi-point thinking tree**, enabling dynamic improvement paths and deeper solution refinement. 

3. **Strong Empirical Results with Comprehensive Evaluation:**  
   The paper presents extensive experimental comparisons across multiple baselines—including deep reasoning models, single-round and multi-round RAG methods—and demonstrates that SolutionRAG achieves *state-of-the-art (SOTA)* performance on both analytical and technical metrics across all eight domains. The ablation studies clearly show the individual contributions of tree-based reasoning and bi-point thinking, while further analysis of tree growth and pruning effectiveness provides convincing evidence of the system’s reasoning capability.
Summary of weaknesses: 1. **Equation Presentation Needs Simplification**

The equations in the paper primarily serve as definitions for data structures or procedural steps (e.g., datapoint formatting, tree growth, node scoring). 

However, many of them are unnecessarily formal or verbose, introducing multiple indices and notation that add cognitive overhead without conveying deeper mathematical insight. 

This detracts from readability and makes the methodology harder to follow, especially for readers outside the core engineering or symbolic reasoning communities. 

Simplifying or rephrasing these definitions in plain language—while retaining key structures—would improve clarity and make the paper more accessible.

2. **Overly Complex Example Figures**

The example figures in the paper, while detailed, contain excessive textual content and tightly packed annotations that can overwhelm the reader. 

Instead of aiding comprehension, the dense text and numerous solution/comment nodes make it difficult to quickly grasp the core idea being illustrated—especially in Figures 1 and 3. 

Simplifying these visuals by reducing on-node text, highlighting only key improvement paths, and offloading detailed explanations to captions or appendices would greatly enhance readability and focus. 

Clearer, more minimal diagrams would make the proposed bi-point thinking and tree exploration processes much easier to understand at a glance.
Comments, suggestions and typos: 1. **Equation Simplification:**  
   Many of the equations in the paper are definitions or procedural representations (e.g., datapoint formats, tree expansion rules, and scoring functions). However, the current mathematical notation is overly formal and complex given their descriptive role. Simplifying the notation or expressing these ideas more narratively would improve accessibility and reduce reader burden, especially for an NLP audience not specialized in symbolic reasoning.

2. **Figure Readability:**  
   Several figures, particularly Figures 1 and 3, are overloaded with dense textual content embedded directly into nodes or branches. This makes them difficult to parse quickly and may discourage readers from fully engaging with the diagrams. It is recommended to simplify the figures—e.g., by reducing text in visual elements, summarizing only key examples, and moving supporting detail to captions or appendices.


3. **Minor Typos / Presentation Notes:**  
   - Line 382: “Logits(us|s(i)_j, c(i+1)_h)” would benefit from a clearer explanation or visual support, especially for those unfamiliar with scoring via logit averaging.  
   - Footnotes and hyperlinks (e.g., marker tool) could be moved to an appendix or formalized for cleaner flow.
Overall assessment scores: 3.5