Thread: A Logic-Based Data Organization Paradigm for How-To Question Answering with Retrieval Augmented Generation

## Abstract

Abstract Recent advances in retrieval-augmented generation have significantly improved the performance of question-answering systems, particularly on factoid ‘5Ws’ questions. However, these systems still face substantial challenges when addressing ‘1H’ questions, specifically how-to questions, which are integral to decision-making processes and require dynamic, step-by-step answers. The key limitation lies in the prevalent data organization paradigm, chunk, which divides documents into fixed-size segments, and disrupts the logical coherence and connections within the context. To overcome this, in this paper, we propose Thread , a novel data organization paradigm aimed at enabling current systems to handle how-to questions more effectively. Specifically, we introduce a new knowledge granularity, termed ‘logic unit’, where documents are transformed into more structured and loosely interconnected logic units with large language models.
Extensive experiments conducted across both open-domain and industrial settings demonstrate that Thread outperforms existing paradigms significantly, improving the success rate of handling how-to questions by 21% to 33%.
Moreover, Thread exhibits high adaptability in processing various document formats, drastically reducing the candidate quantity in the knowledge base and minimizing the required information to one-fourth compared with chunk, optimizing both efficiency and effectiveness.

## 1 Introduction

Question answering (QA) is a foundational research topic in human-machine interaction (Allam & Haggag, ).
Among the most advanced techniques, Retrieval-augmented generation (RAG) organizes external documents into fixed-size chunks and retrieves relevant knowledge to enhance QA systems (Shao et al., ; Trivedi et al., ; Jiang et al., ; Asai et al., ).
This routine is particularly effective in handling the ‘5Ws’ questions, which are typically factoid requiring factual information about specific details (Yang et al., ; Jiang et al., ; Kwiatkowski et al., ; Stelmakh et al., ), such as ‘When is Shakespeare’s birthday?’.
Specifically, the data organization paradigm only needs to provide chunks containing the relevant knowledge, *e.g.* triples or documents about the topic entity, to the RAG systems to facilitate the ‘5Ws’ questions. However, the ‘1H’ questions, derived from Aristotle’s Nicomachean Ethics (in this paper as ‘how-to’ questions, Crisp ), remain largely underexplored(^1^11The ‘5Ws’ represent What, Why, When, Where, and Who, and the ‘1H’ stands for How. The ‘Why’ question can sometimes delve into explanations and motivations.).
These questions are in high demand in practical applications such as teaching us how to write code to achieve specific goals.

Central to problem-solving (Polya & Pólya, ) and human learning in cognitive science (Learn, ), how-to questions are inherently more complex, often involve processes that require interpretation and analysis (Deng et al., ).
For example, correctly answering the question shown in Figure 1 ‘How to diagnose and fix a performance issue in a web application?’ involves a multi-step decision-making process, *i.e.*, firstly checking the server load and response time, then optimizing server configuration or indexes and queries with user feedback. Such a dynamic nature necessitates the RAG systems incrementally guide users through each step, adapting to specific circumstances and providing precise and logical information.
However, the prevalent data organization paradigm, chunk(^2^22‘Chunk’ refers to a general document splitting paradigm including chunks, sentences, phrases, etc.) (Splitter, ; Chen et al., ; Gao et al., ) leaves current RAG systems (Asai et al., ; Shao et al., ) struggling to solve such questions. This chunk-based paradigm disrupts the logical flow and coherence of the content due to its inability to effectively represent the internal logic. Consequently, it returns excessive information during inference, making it difficult to maintain continuity between sequential steps, leading to fragmented responses for how-to questions and posing significant challenges for users.
Overcoming these limitations necessitates a paradigm shift that accounts for the logical structure and stepwise nature of how-to questions, enabling more precise and coherent information retrieval tailored to specific needs.

Figure: Figure 1: An example of how-to questions with its decision-making process. We omit details such as the actions to check the server load and response time due to limited space.
Refer to caption: x1.png

In this paper, we propose Thread, a new logic-based data organization paradigm designed to enhance the handling of how-to questions. The name Thread evokes the idea of ‘Pulling on the thread, the whole mystery started to unravel like a sweater.’ (Garcia & Stohl, ). Specifically, we introduce a new knowledge granularity named ‘logic unit’, comprising five key components and four different types (see §3.1 and §3.2). We employ a two-stage process depicted in §3.3 to extract logic units (LUs) from documents. The first, optional stage is reformulating the original documents depending on their format and style, and the second focuses on extracting and merging logic units. In this way, Thread captures connections within the documents, breaking them into more structured and loosely interconnected logic units.
When answering how-to questions, the system integrated with Thread enables a dynamic interaction manner. First, it retrieves relevant LUs based on their indexed headers. Then, the body of the selected LU provides the necessary content to generate responses for the current step. With user feedback, the linker in LU dynamically connects to other LUs, allowing the system to adapt its responses until the how-to question is comprehensively addressed.

We evaluate the effectiveness of Thread by conducting experiments in two open-domain, web navigation (Deng et al., ) and Wikipedia Instructions (Koupaee & Wang, ), and one industrial setting, Incident Mitigation (Shetty et al., ). Experimental results demonstrate that while existing paradigms struggle with how-to questions, Thread excels at handling consecutive steps and consistently outperforms them, especially in real-world incident mitigation scenarios, with an improvement of Success Rate ranging from 21.05% to 33.33%. In addition, Thread demonstrates great efficiency and effectiveness in processing various document formats, significantly reducing the number of retrieval units to 20% and the length of tokens needed for generation to 22.60%.
The main contributions of this paper include:

- •
We highlight the challenges faced by current RAG systems in addressing how-to questions. To address the limitation of the chunk-based paradigm, we propose Thread, a novel data organization paradigm that transforms original documents into structured, interconnected logic units.
- •
Integrated with Thread, our system follows a dynamic interaction mode, guiding users incrementally through each step and adapting to their specific circumstances. Our system also brings more possibilities for an automation pipeline, solving how-to questions more efficiently.
- •
Experimental results show that Thread significantly outperforms existing data organization paradigms across three scenarios. Additionally, Thread requires less information, while effectively handling various document formats and substantially reducing the knowledge base size.

## 2 Related Work

#### Data Organization Paradigm in RAG

The data organization process is a critical pre-stage of RAG methods where documents are processed and segmented following certain data organization paradigms. The most common data organization paradigm is splitting documents into retrieval units (Gao et al., ). These retrieval units vary in granularity such as phrases, sentences, propositions (Chen et al., ), chunks (Kamradt, ), etc. Coarser-grained units contain more information but introduce redundant noise, while finer-grained units have lower semantic integrity and often require retrieving more units to gather comprehensive information. However, the chunk-based data organization paradigm ignores the logical and relational connections between chunks, potentially disrupting the inherent logic flow in documents. Another paradigm constructs documents into knowledge graphs (KG), where retrieval units include entities, triplets, etc. (Gaur et al., ; Sen et al., ; He et al., ; Wang et al., ; Edge et al., ). These approaches emphasize semantic/lexical similarities between retrieval units, their success in factoid questions is limited when applied to how-to questions. This limitation arises because how-to questions demand logical connections between retrieval units that extend beyond mere semantic or lexical similarities.

#### Information Retrieved by RAG

The effectiveness of RAG methods depends on the generator’s ability to utilize retrieved information and the quality and quantity of that information. Insufficient question-relevant information can cause hallucinations in LLM-based generators (Li et al., ; Zhang et al., ), making it crucial to improve the retrieval process. Traditional one-round retrieval methods (Guu et al., ; Lewis et al., ) often fail to gather all necessary information due to their reliance on the similarity between query and retrieval units (Gan et al., ).
Advanced RAG methods use query rewriting and expansion (Shao et al., ; Trivedi et al., ; Kim et al., ) or iterative retrieval (Shao et al., ; Jiang et al., ; Asai et al., ) to collect more information. However, these approaches still struggle with how-to questions, which require making next-step decisions based on the current retrieved units, unless the current retrieved units contain clues that lead to the next step. The main issue is the lack of connections between retrieval units, which prevents effective retrieval and the gathering of sufficient information.

## 3 Methodology

In this section, we introduce ‘logic unit’ with its components and types, then explain the process of constructing the Thread knowledge base, followed by an illustration of how Thread integrates with existing systems.

### 3.1 Logic Unit: Retrieval Unit of Thread

We propose a new knowledge granularity called ‘logic unit’. Different from the chunk-based paradigm, LU comprises specially designed components, especially for bridging consecutive steps when addressing how-to questions, considering the internal logic and coherence of documents.

Prerequisite.
The prerequisite component acts as an information supplement, providing the necessary context to understand the LU. For example, an LU may include domain-specific terminology such as entities or abbreviations.
The prerequisite explains these terms and can generate new queries to retrieve LUs with more detailed information. Without this context, passing these LUs to an LLM-based generator could lead to hallucinations. Additionally, the prerequisite can function as an LU filter, containing constraints that must be met before the LU is considered in answer generation. This filtering ensures only relevant LUs are retrieved. As shown in step LU in Figure 2, accessing the server monitor is a prerequisite for checking the server load.

Header.
The header summarizes the LU or describes the intention it aims to address, depending on the type of LU (refer to §3.2).
For example, the header could be the name of a terminology if LU describes a terminology; if the LU describes actions to resolve a problem, the header describes the intent or the problem LU aims to resolve. Different from chunk that indexes the entire content, we use the header for indexing which serves as the key for retrieving the LU based on a query.

Body.
The body contains detailed information on the LU, which is the core content fed into the LLM-based generator to generate answers. It includes specific actions or necessary information such as code blocks, detailed instructions, etc. This detailed content helps resolve the query mentioned in the header or provides a detailed explanation of the header.

Linker.
The linker acts as a bridge between logic units, enabling the dynamic process of how-to questions. Unlike the chunk-based paradigm, which relies on previous retrieval units that often lack direct clues, the linker in Thread provides necessary information to generate new queries for subsequent retrieval. In Figure 2, the linker of ‘checking server’ specifies multiple possibilities after taking the action in LU body, guiding the retrieval of the next-step LU. Its format varies by LU type, serving as either a query for retrieving other LUs or an entity relationship. The edge of knowledge graph in traditional factoid questions is a special linker enabling navigation between related entities. When no further LUs are connected, the linker remains empty, isolating the current LU.

Meta Data.
The meta data includes information about the source document from which the LU is extracted, such as the document title, ID, date, and other relevant details. This meta data is crucial for updating LUs when the source documentation is revised and reprocessed.

Figure: Figure 2: The construction process of Thread. The left part depicts the two-stage process including reformulating documents into structured ones and extracting and merging logic units, the right part instantiates the extraction by providing an example.
Refer to caption: x2.png

### 3.2 Logic Unit Type

When converting documents into logic units, we distinguish how-to questions into ‘linear’ and ‘dynamic’ types, where linear how-to questions often involve a fixed sequence of steps
that do not require feedback or decision points based on intermediate outcomes.
Below are the common LU types identified in our experiments(^3^33These LU types are summarized from our practice in experiments with industrial and public datasets. There may be additional LU types depending on the specific scenario.), showing Thread is versatile enough to handle diverse types:

Step.
This is the most common LU type for resolving how-to questions. Each LU body represents detailed actions, including code blocks and resolution instructions. The LU prerequisite describes the actions that need to be completed before executing the current actions. The prerequisite is especially crucial for identifying the entry point of a solution. For example, when facing a problem, there exist different ways to resolve it depending on the current situation. The prerequisite serves as a condition in the LU selection stage, filtering out LUs that do not meet the prerequisite.

Terminology.
This type provides detailed explanations of domain-specific terminology. For example, terms may share the same name or abbreviation in the LU header but convey different meanings in the LU body. The prerequisites for terminology LUs describe scenarios where the terminology typically appears and linkers are usually empty unless referencing extended terminology that depends on it.

FAQ.
This type provides frequently asked questions, supplementing the knowledge base. These LUs are typically isolated, with the LU body offering solutions through sequential steps that address linear how-to questions not reliant on dynamic states. They save time by avoiding the need for sequentially retrieving LUs for common questions.

Appendix.
This type provides additional information relevant to the scenario of LUs, such as examples, background, lookup tables, etc. These LUs serve as supplementary knowledge for LLMs when generating responses or executable plans.

### 3.3 Thread: LU-based Knowledge Base

In practice, documentation is often unstructured and varies in format and style. Our approach to converting documentation into Thread involves a two-stage process to obtain LUs, as shown in the left part of Figure 2. We believe that with advancements in LLMs, this process could be streamlined into a single stage, enabling high-fidelity LU conversion with minimal hallucination.

Documentation Reformulation (Optional).
This stage is optional, depending on the quality of the documentation. For example, in software engineering, Troubleshooting Guides (TSGs) often vary in format, include diverse types of information, and lack readability and detail, negatively impacting productivity and service health (Shetty et al., ). Due to such format, where some are clearly outlined and others are disordered, we avoid directly extracting LUs from original documents. Instead, we first reformulate these documents into structured formats. Leveraging LLMs for this task, we enhance the LLMs’ in-domain understanding by providing search capabilities and domain-specific context. This is followed by a refinement step to prevent overlooking details or hallucinating information. Figure 2 (a) shows the reformulation stage and the right represents with symbols.
But it is unnecessary for well-written documents like product help docs, which typically follow a linear how-to format. We illustrate the process in Table LABEL:tab:example_lu by example and provide prompts in Appendix B.

LU Extraction and Merge.
After reformulation, multiple LUs of varying types can be extracted from a single structured document (shown in Figure 2 (b)). Unlike chunk-based data organization commonly with fixed chunk sizes, LU granularity depends on content. For example, solutions to linear how-to questions typically form a single path from start to completion, with interconnected steps and no multiple execution outcomes encapsulated in one LU, such as an FAQ LU. However, for dynamic how-to questions with multiple possible outcomes, it is better to have one step per LU (Step LU), with Linkers navigating to the next LUs. Note that in dynamic how-to questions, not every step has multiple execution outcomes. If only one next step exists, the LUs can be merged to include both current and subsequent steps. Additionally, LUs with similar Headers and Bodies should be merged, extending the Prerequisite and Linker.

LU update.
In industry, documentation is often updated with each product version release. When this happens, we redo the above steps for the updated documentation, identifying LUs in Thread with their Meta Data and replacing outdated LUs.

As we extract and merge LUs, the collection of LUs from all documents forms the knowledge base, Thread. This LU-based knowledge base serves as an essential component compatible with the current RAG system and even makes it possible for automation.

### 3.4 Integrate Thread with QA System

To demonstrate how Thread works, we use dynamic how-to questions as an example. Figure 3 shows the RAG system incorporating our Thread data organization paradigm. The Retriever and LLM-based Generator are inherited from the original RAG system.
LUs are indexed by their Headers. When an initial how-to question is submitted, the Retriever identifies the top-K most relevant LUs based on query-header similarity. The Selector then checks the prerequisites of these LUs and filters out those that do not meet the current prerequisites, derived from the initial question or any available chat history. If no current prerequisite is provided, the system can prompt the user, *e.g.*, ‘Before doing …, have you tried …?’, to obtain the current prerequisite for LU filtering.
After selection, the body of the LUs is fed into the LLM-based generator to produce an answer. If an execution engine is available, actions can be executed automatically; otherwise, the answer/action is presented to the user for manual execution. Once the action is executed, the Linker matches one of the possible outcomes and generates a new query for the next retrieval round.

Unlike traditional RAG systems, Thread-enabled systems can be potentially fully or semi-automated when integrated with execution engines. This integration offers greater automation and flexibility, as updating LUs automatically updates the system, compared to manually designed pipelines (see §3.3).

Figure: Figure 3: The RAG system integrated with Thread. It retrieves relevant LUs based on query-Header similarity and filters out LUs that do not meet the current prerequisites. The selected LUs are passed to LLMs to generate actions for execution. After execution, the Linker matches results and generates a new query for the next retrieval iteration.
Refer to caption: x3.png

## 4 Experimental Setup

### 4.1 Scenarios and Datasets

We evaluate Thread on two open-domain scenarios: Web Navigation (Mind2Web (Deng et al., )) and Wikipedia Instructions (WikiHow (Koupaee & Wang, )), and one industrial setting: Incident Mitigation (IcM (Shetty et al., ; An et al., )).
We provide one example of each dataset in Appendix B to better demonstrate linear and dynamics how-to questions.

Web Navigation.
Mind2Web (Deng et al., ) is a dataset designed for web agents to perform complex tasks on real-world websites based on language instructions. Each task involves a ‘dynamic how-to question’, with multiple possible outcomes depending on the state of the executed actions.

Wikipedia Instructions.
WikiHow(^4^44[https://www.wikihow.com](https://www.wikihow.com)) is a platform containing numerous articles that offer step-by-step guidance on various procedural tasks. Each article is typically titled ‘How to’ and includes a brief task description, followed by a linear, fixed sequence of steps.

Incident Mitigation.
Incident Mitigation (Shetty et al., ; An et al., ; Jiang et al., ) is essential for operating large-scale cloud services, where engineers use Troubleshooting Guides (TSGs) to address incidents. Each step in incident mitigation can yield different outcomes depending on the state, making it suitable for testing Thread on ‘dynamic how-to questions’. Unlike the other open-domain datasets, we perform a human evaluation involving twenty on-call engineers (OCEs) responsible for incident mitigation. We collect five incidents, classified into two simple and three hard ones based on their mitigation steps in history. Each OCE is tasked with mitigating all five incidents, using randomly ordered baselines per incident to avoid familiarity bias. The RAG system initiates automated mitigation for each incident; if it encounters a failure at any step, an OCE intervenes to address the issue before the system resumes automated procedures(^5^55Varying across new-hire and experienced OCEs. Mitigating one incident costs each OCE around 30 minutes to 1 hour manually, and we collect five incidents to ensure consistency, engagement, and ethical treatment. Note that one OCE’s data was contaminated during the experiment, so we removed that OCE’s data.).

### 4.2 Documents for Retrieval

**Table 1: The statistics of datasets, including the number of documents, derived chunks, LUs, etc.**
| Dataset | #Docs | #Tasks | #Steps | #Chunks | #LUs |
| --- | --- | --- | --- | --- | --- |
| Mind2Web | 490 | 252 | 2094 | 6210 | 1089 |
| WikiHow | 97 | 97 | 2140 | 4225 | 774 |
| IcM | 56 | 95 | 323 | 413 | 378 |

As the Mind2Web dataset lacks relevant documents, we create retrieval documents tailored to the RAG system. Assuming that each website has help docs applicable across different tasks, we use the "Cross-Task" test set, selecting examples from the training set to craft informative documents for each website. And we follow Wang et al. () to brainstorm different formats of documents. More details about collecting documents are shown in Appendix C.2.

For the WikiHow dataset, we utilize publicly available Windows Office Support Docs(^6^66[https://github.com/MicrosoftDocs/OfficeDocs-Support](https://github.com/MicrosoftDocs/OfficeDocs-Support)) as retrieval documents. We select around 100 tasks from WikiHow tagged with Microsoft products like Word, PowerPoint, and Teams, each containing 10 to 40 steps related to Windows operations.

For the IcM dataset, we collect 56 TSGs from an enterprise-level engineering team responsible for a large-scale cloud platform. The selected incidents in the IcM dataset can be resolved using the knowledge provided in these TSGs. We list the statistics of our datasets in Table 1.

### 4.3 Baselines

In the Mind2Web dataset, previous work has not treated it as how-to questions. State-of-the-art methods like SYNASE (Zheng et al., ) and MINDACT (Deng et al., ) either use In-Context Learning (ICL), providing few-shot demonstrations, or Supervised Learning (SL) to finetune a model on the training set. To ensure a fair comparison with our LLM endpoints(^7^77We use GPT-3.5 and GPT-4 with version 1106-preview.), we re-implement the MINDACT experiments with the same demonstrations and included chat history as extra context.
In our paper, we treat the Mind2Web dataset as dynamic how-to questions and use our RAG system to solve these tasks. For comparison, we use doc-based (providing the entire document) and chunk-based data organization paradigms as baselines against Thread. This RAG system with different data organization paradigms also serves as baselines for the WikiHow and IcM datasets.
More details about experiments can be found in Appendix LABEL:appendix_exp, D.

### 4.4 Evaluation Metrics

For the Mind2Web dataset, each task is treated as a multi-choice question. We adapt the evaluation metrics used in (Deng et al., ), which include: Element Accuracy (Ele. Acc) to compare the chosen HTML element with all provided elements; Operation F1 (Op. F1) to calculate the token-level F1 score for predicted operations such as “CLICK”, “TYPE IN”, etc.; Step Success Rate (Step SR), where a step is successful if both the selected element and predicted operation are accurate; and Success Rate (SR), where a task is successful only if all steps are successful.

For the WikiHow dataset, which contains ground truth steps, we leverage LLMs to extract “Action Items” from each ground truth step and generated step, and we use the following metrics: Precision ($P=\frac{\#matched_items}{\#total_generated_items}$); Recall ($R=\frac{\#matched_items}{\#total_groundtruth_items}$); F1; and Success Rate (SR) to assess if the generated steps can successfully complete the task, using LLMs to evaluate (Appendix C.3 shows the evaluation prompt).

For the IcM dataset, which involves task execution, we perform evaluations with OCEs (refer to §4.1) using five metrics: Success Rate (SR) indicating the percentage of incidents mitigated automatically by the system without human intervention; Step Success Rate (Step SR) representing the percentage of successful steps out of all task steps; Pre-Failure Step Success Rate (P.F. Step SR) representing the percentage of successful steps before the first failure; Human Intervention (HI) measuring the percentage of steps requiring human intervention; and Average Turns (Turns) to measure the average interaction turns between OCEs and the system during incident mitigation.

## 5 Experimental Results

### 5.1 Main Results

Web Navigation.
The overall performance on Mind2Web is shown in Table 2, where we compare our method with baselines and both doc-based and chunk-based RAG methods. We observe that providing informative documents (regardless of the data organization paradigm) within RAG methods significantly improves performance. RAG methods outperform ICL and SL methods. By incorporating external documents, the doc-based RAG method achieves performance comparable to the best results of MINDACT. Notably, our Thread paradigm is the best among RAG methods, showing improvements of 4.06% in Ele. Acc, 3.49% in Step SR and 3.57% in SR. Note that MINDACT-SL gets the highest Op. F1 due to label distribution imbalance(^8^88If the model predicts all operations as ”CLICK,” the Op. F1 would reach 79.90%.), leading the model to favor generating the most frequent operations.

**Table 2: Experiment results on Mind2Web. ‘*’ represents taking results from the original paper.**
|  |  |  | Mind2Web Cross-Task |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method | Model | Paradigm | Ele. Acc | Op. F1 | Step SR | SR |
| SYNASE () | w/ GPT-3.5* | ICL | 34.00 | - | 30.60 | 2.40 |
|  | w/ GPT-3.5 | ICL | 40.69 | 49.66 | 33.91 | 1.59 |
|  | w/GPT-4 | ICL | 62.80 | 60.37 | 51.81 | 10.32 |
| MINDACT () | w/ Flan-T5XL* | SL | 55.10 | 75.70 | 52.00 | 5.20 |
|  | w/ GPT-4 | Chunk | 64.23 | 65.96 | 58.45 | 8.73 |
|  | w/ GPT-4 | Doc | 63.80 | 65.89 | 58.36 | 11.51 |
| RAG | w/ GPT-4 | Thread | 68.29 | 69.53 | 61.94 | 12.30 |

**Table 3: Experiment results on Incident Mitigation. We divide incidents into two groups: simple or hard, and compare the performance separately.**
|  |  | Incident Mitigation |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Paradigm | Category | SR | Step SR | P.F. Step SR | HI | Turns |
|  | Simple | 40.51 | 60.90 | 60.90 | 30.10 | 3.14 |
| Chunk | Hard | 28.95 | 53.16 | 43.05 | 46.84 | 6.84 |
|  | Simple | 43.86 | 63.90 | 63.90 | 36.09 | 2.98 |
| Doc | Hard | 31.58 | 57.89 | 42.11 | 42.11 | 6.53 |
|  | Simple | 77.19 | 88.72 | 84.21 | 11.28 | 2.56 |
| Thread | Hard | 52.63 | 84.21 | 68.95 | 15.79 | 5.74 |

Incident Mitigation.
Table 3 illustrates the advantages of Thread over other paradigms when addressing complex dynamic how-to questions. Both chunk-based and doc-based RAG methods exhibit significant limitations in incident mitigation, as their SR and P.F. Step SR are much lower due to their inability to connect subsequent steps based on the current step. In contrast, our Thread achieves the best score across all metrics, particularly achieving a significant increase in SR from 21.02% to 33.33%. More importantly, the highest P.F. Step SR highlights Thread’s ability to dynamically connect to subsequent steps based on user feedback, thereby reducing the need for human intervention. Consequently, Thread not only achieves the highest Step SR but also requires the fewest interaction turns to effectively mitigate both simple and complex incidents.

**Table 4: Experiment results on WikiHow with different interaction manners and paradigms.**
|  | WikiHow |  |  |  |
| --- | --- | --- | --- | --- |
| Paradigm | SR | P | R | F1 |
| Single-Turn |  |  |  |  |
| Chunk | 19.59 | 60.20 | 25.16 | 35.49 |
| Doc | 58.76 | 77.71 | 57.93 | 66.37 |
| Thread | 63.91 | 83.43 | 71.48 | 76.99 |
| Multi-Turn |  |  |  |  |
| Chunk | 20.62 | 52.95 | 25.54 | 34.45 |
| Doc | 68.04 | 87.10 | 70.65 | 78.02 |
| Thread | 72.16 | 89.77 | 73.36 | 80.74 |

Wikipedia Instructions.
Table 4 presents the performance of various data organization paradigms on WikiHow tasks, experimenting with both single-turn and multi-turn interactions. In the single-turn setting, where the RAG system needs to generate the entire plan in a single interaction, our Thread outperforms the doc-based method with 5.15%. This indicates that Thread delivers information with less redundancy while achieving a higher SR score. Additionally, when comparing Thread to the chunk-based method, we observe a significantly lower success rate (SR) of 19.59%. This result highlights the challenge of retrieving all relevant chunks without considering the logic flow inside documents.

In the multi-turn setting, the RAG system performs iterative retrieval, showing superior performance compared to single-turn. This indicates the advantage of a step-by-step approach in handling how-to questions. We can see that the SR improves significantly from 58.76% to 68.04% in the doc-based paradigm and from 63.91% to 72.16% in Thread. Similar to the single-turn setting, the chunk-based method splits documents while disrupting internal logic, resulting in a low SR of 20.62%. Our Thread excels across all metrics, achieving the highest SR of 72.16%, highlighting its effectiveness in maintaining and modeling the connections between steps.

### 5.2 Ablation on RAG System Settings

This section presents an ablation study on the key settings of our RAG system. While retriever and generator variants have been explored in other research (Gao et al., ), we use text-embedding-ada-002 (OpenAI, ) for the retriever and GPT-4 as the LLM-based generator. The ablation study is performed on Mind2Web dataset.

Multi-turn Interaction. As shown in Table 4, the multi-turn setting is better than the single-turn setting in answering how-to questions. We use multi-turn setting for the other scenarios in this paper.

Chat History. The chat history helps the system to check previous actions and results, helping the system make better decisions. Without chat history, the performance drops as shown in Table 5. We include chat history for all RAG-based methods in our experiments.

**Table 5: Ablation study of integrating chat history and retrieval unit selection on Mind2web.**
| Paradigm | Ele. Acc | Op. F1 | Step SR |
| --- | --- | --- | --- |
| ICL | 62.80 | 60.37 | 51.81 |
| $w/o.$ historical steps | 56.97 | 59.09 | 50.38 |
| Chunk | 59.94 | 62.58 | 54.39 |
| $w/o.$ chunk selection | 64.23 | 65.96 | 58.45 |
| Thread | 68.29 | 69.53 | 61.94 |
| $w/o.$ LU selection | 67.05 | 68.43 | 60.79 |

Retrieval Units Selector.
As aforementioned in §3.4 and Figure 3, the Selector picks the most relevant retrieval units from the top-K retrieved units. Table 5 shows the ablation of the retrieval unit selector. Comparing chunk selection to LU selection, the performance of Ele. Acc drops by 4.29% and Op.F1 by 3.38%. Conversely, the selector improves all metrics with applied to Thread. Unlike LU selection, which filters out irrelevant LUs based on prerequisites, chunk selection disregards inter-chunk connections and may exclude relevant chunks. Therefore, we activate the selector only for our Thread.

### 5.3 Analysis of Data Organization Paradigms

We compare various data organization paradigms within RAG, implementing Semantic Chunking (Kamradt, ) and Proposition (Chen et al., ) in addition to recursive chunking (Splitter, ) (chunk-based) and entire document (doc-based) approaches (details in Appendix A). Table 7 shows that our Thread paradigm outperforms the other paradigms across all metrics. Thread w/o. represents without LU selector. Although Semantic and Proposition methods use LLMs to merge semantically similar sentences, they fail to adequately address logical connections, resulting in poor performance. Additionally, our system processes the smallest token length of retrieval units yet achieves the highest performance, underscoring our approach’s efficiency and effectiveness.

Figure: Figure 4: Analysis of using different LLMs on WikiHow (GPT-4 v.s. LLaMA3-70B).
Refer to caption: x4.png

**Table 6: Analysis of data organization paradigms.**
| Paradigm | Ele. Acc | Op. F1 | Step SR | #Tokens in RU |
| --- | --- | --- | --- | --- |
| Doc | 63.80 | 65.89 | 58.36 | 663.84 |
| Recursive | 64.23 | 65.96 | 58.45 | 695.77 |
| Semantic | 65.14 | 67.30 | 59.93 | 1337.16 |
| Proposition | 62.37 | 64.78 | 56.78 | 790.14 |
| Thread w/o. | 67.05 | 68.43 | 60.79 | 772.67 |
| Thread | 68.29 | 69.53 | 61.94 | 157.10 |

### 5.4 Document Formats in LU Extraction

As mentioned in §4.2, we test our LU extraction method in accommodating varying document structures. We generate different document formats (detailed in Appendix C.2), including structured markdown, hierarchical guidelines, tabular checklists, and narrative documents. Table 7 shows that our Thread paradigm effectively organizes these formats, consistently outperforming the chunk-based paradigm across all metrics. Our Thread improves Ele. Acc by up to 9.61%, Op. F1 by up to 8.43%, and Step SR by up to 8.51%. Especially, our Thread achieves the highest performance with structured documents, as this format helps construct a higher-quality knowledge base.

### 5.5 Different LLMs as backbone

We further conduct experiments on LLaMA3-70B, as shown in Figure 4. Although LLaMA3-70B is less powerful than GPT-4, it still demonstrates competitive performance with the help of Thread. Results indicate that LLaMA3 struggles with WikiHow questions applying chunk-based or doc-based paradigms. However, with the integration of Thread, LLaMA3 not only achieves much better performance in SR, but also narrows the gap with GPT-4. Specifically, LLaMA3 achieves SR and F1 scores of 60.82% and 72.84%, respectively, compared to GPT-4’s scores of 72.16% and 80.74%.
This indicates that while a performance gap remains, particularly with the chunk-based and doc-based paradigms, our Thread considerably reduces this disparity between GPT-4 and LLaMA3. These findings highlight the value of our proposed paradigm in enhancing the performance of different LLMs, showcasing its generalizability, robustness, and efficiency in handling how-to questions.

## 6 Conclusion

In this paper, we address the overlooked category of handling how-to questions in QA systems by proposing Thread, a novel data organization paradigm that captures logical connections within documents. By introducing a new knowledge granularity called ’logic unit,’ Thread restructures documents into interconnected logic units that are compatible with RAG methods. Extensive experiments show that Thread significantly outperforms existing paradigms, improving performance while reducing the knowledge base size and minimizing the information needed for generation.

## Appendix A Current Data Organization Paradigm

From Gao et al. (), current data organization paradigms can be categorized into phrases, sentences, propositions, chunks, and so on.
In our paper, we choose chunks (^9^99We use the implementation by LangChain [https://python.langchain.com/v0.2/](https://python.langchain.com/v0.2/)) and propositions to compare with our proposed Thread (^10^1010Note: For chunks, we retrieve the top-5 at each time, and for documents, we only retrieve the top-1.).

Recursive Chunk.
This chunking method splits the original documents using a list of separators, then reassembles them according to specified chunk sizes and overlap sizes. In our experiment, we use different chunk sizes for each dataset: 1000 for Mind2Web, 2000 for IcM, and 300 for WikiHow. The chunk overlap sizes also vary 50 for Mind2Web, 100 for IcM, and 30 for WikiHow.

Entire Document.
This method sends the entire document directly into the model, constrained by the document’s length and structure.

Semantic Chunk. Kamradt () proposes splitting chunks based on semantic similarity. The hypothesis is that semantically similar chunks should be grouped together. By comparing the semantic similarity between adjacent sentences, the method identifies “break points”. If the similarity in the embedding space exceeds a certain threshold, it marks the start of a new semantic chunk.

Agentic Chunk (Proposition). Chen et al. ()(^11^1111For proposition paradigm, we use agentic chunker since the input token of Flan-T5 is limited to 512, [https://github.com/FullStackRetrieval-com/RetrievalTutorials](https://github.com/FullStackRetrieval-com/RetrievalTutorials).) introduces the concept of the Proposition Paradigm, which involves extracting independent propositions from original documents. The Agentic Chunk method is based on this paradigm. It first splits the documents into paragraphs, then extracts propositions from each paragraph, and at last merges similar propositions into chunks.

In all our experiments, we set the temperature of LLMs to 0 and top_p to 1 for results reproduction.

## Appendix B Example of Dataset

We list one example of each dataset in Table B, and an example of the construction process of Thread including the original document, reformulated document, and its corresponding logic unit in Table LABEL:tab:example_lu.

**Table 8: Examples of each dataset. For Mind2Web, although the test set has fixed options for each step, there are different execution methods for the same task on each website, so it is essentially dynamic.**
| Dataset | Example |
| --- | --- |
| Mind2web<br>(Dynamic) | [⬇](data:text/plain;base64,PGh0bWw+IC4uLiA8L2h0bWw+CkJhc2VkIG9uIHRoZSBIVE1MIHdlYnBhZ2UgYWJvdmUsIHRyeSB0byBjb21wbGV0ZSB0aGUgZm9sbG93aW5nIHRhc2sKVGFzazogQm9vayB0aGUgbG93ZXN0LXByaWNlZCBhbmQgcXVpY2tlc3QgZmxpZ2h0IGZvciA1IGFkdWx0cyBhbmQgMSBjaGlsZCBvbiBNYXkgMjAgZnJvbSBNdW1iYWkgdG8gYW55IGFpcnBvcnQgbmVhciBXYXNoaW5ndG9uLgpQcmV2aW91cyBhY3Rpb25zOgpOb25lCldoYXQgc2hvdWxkIGJlIHRoZSBuZXh0IGFjdGlvbj8KUGxlYXNlIHNlbGVjdCBmcm9tIHRoZSBmb2xsb3dpbmcgY2hvaWNlcyAoSWYgdGhlIGNvcnJlY3QgYWN0aW9uIGlzIG5vdCBpbiB0aGUgcGFnZSBhYm92ZSwgcGxlYXNlIHNlbGVjdCBBLiAnTm9uZSBvZiB0aGUgYWJvdmUnKToKQS4gTm9uZSBvZiB0aGUgYWJvdmUKQi4gPGRpdiBpZD0wPiA8aW5wdXQgcmFkaW8gdHJpcHR5cGUgcm91bmR0cmlwIHRydWUgLz4gPGxhYmVsPiA8c3Bhbj4KQy4gPGxhYmVsIGlkPTE+IDxzcGFuPiBTZWFyY2ggZmxpZ2h0cyBvbmUgd2F5IDwvc3Bhbj4gPHNwYW4+IE9uZQpELiA8YSBpZD0yPiA8aDM+IENlbGVicmF0ZSBXb3JsZCBXaXNoIERheSA8L2gzPiA8cD4gU3VwcG9ydApFLiA8aDIgaWQ9Mz4gSGVscCA8L2gyPgpGLiA8YSBpZD00PiA8aW1nIGZvbGxvdyB1cyBvbiB0d2l0dGVyIC0gLz4gPC9hPgpDLiAgQWN0aW9uOiBDTElDSw==)<br><html> ... </html><br>Based on the HTML webpage above, try to complete the following task<br>Task: Book the lowest-priced and quickest flight for 5 adults and 1 child on May 20 from Mumbai to any airport near Washington.<br>Previous actions:<br>None<br>What should be the next action?<br>Please select from the following choices (If the correct action is not in the page above, please select A. ’None of the above’):<br>A. None of the above<br>B. <div id=0> <input radio triptype roundtrip true /> <label> <span><br>C. <label id=1> <span> Search flights one way </span> <span> One<br>D. <a id=2> <h3> Celebrate World Wish Day </h3> <p> Support<br>E. <h2 id=3> Help </h2><br>F. <a id=4> <img follow us on twitter - /> </a><br>C. Action: CLICK |
| WikiHow<br>(Linear) | [⬇](data:text/plain;base64,IlByb2JsZW0iOiAiSG93IHRvIEFkZCBDYXB0aW9ucyB0byBUYWJsZXMgaW4gTWljcm9zb2Z0IFdvcmQiLAoiU29sdXRpb24gU3RlcHMiOiBbCiAgICAiU2VsZWN0IHRoZSB0YWJsZSB0byB3aGljaCB5b3Ugd2FudCB0byBhZGQgYSBjYXB0aW9uLiIsCiAgICAiVXNpbmcgeW91ciBtb3VzZSwgY2xpY2sgYW5kIGRyYWcgb3ZlciB0aGUgZW50aXJlIHRhYmxlIHRvIHNlbGVjdCBpdC4iLAogICAgIlJpZ2h0LWNsaWNrIChvciBjdHJsLWNsaWNrKSB0aGUgdGFibGUgYW5kIHNlbGVjdCBJbnNlcnQgQ2FwdGlvbi4iLAogICAgIkVudGVyIHlvdXIgY2FwdGlvbi4iLAogICAgIlR5cGUgdGhlIGNhcHRpb24gZm9yIHRoaXMgdGFibGUgaW50byB0aGUgXCJDYXB0aW9uXCIgZmllbGQuIiwKICAgICJTZWxlY3QgYSBjYXB0aW9uIGxhYmVsLiIsCiAgICAiQ3VzdG9taXplIHlvdXIgY2FwdGlvbiBudW1iZXJzIChvcHRpb25hbCkuIiwKICAgICJDaG9vc2Ugd2hlcmUgdG8gcGxhY2UgeW91ciBjYXB0aW9uLiIsCiAgICAiQ2xpY2sgdGhlIFwiUG9zaXRpb25cIiBkcm9wLWRvd24gbWVudSwgYW5kIGNob29zZSB3aGV0aGVyIHRvIHBsYWNlIHRoZSBjYXB0aW9uIGFib3ZlIG9yIGJlbG93IHRoZSB0YWJsZS4iLAogICAgIkNsaWNrIE9LIHRvIGFkZCB5b3VyIGNhcHRpb24gdG8gdGhlIHRhYmxlLiIsCiAgICAiRm9ybWF0IHlvdXIgY2FwdGlvbnMuIgpd)<br>"Problem": "How to Add Captions to Tables in Microsoft Word",<br>"Solution Steps": [<br>"Select the table to which you want to add a caption.",<br>"Using your mouse, click and drag over the entire table to select it.",<br>"Right-click (or ctrl-click) the table and select Insert Caption.",<br>"Enter your caption.",<br>"Type the caption for this table into the \"Caption\" field.",<br>"Select a caption label.",<br>"Customize your caption numbers (optional).",<br>"Choose where to place your caption.",<br>"Click the \"Position\" drop-down menu, and choose whether to place the caption above or below the table.",<br>"Click OK to add your caption to the table.",<br>"Format your captions."<br>] |
| IcM<br>(Dynamic) | [⬇](data:text/plain;base64,SG93IHRvIEludmVzdGlnYXRlIFNlcnZpY2UgQS1Uby1TZXJ2aWNlIEIgQ29ubmVjdGlvbj8KCiMjIyBTdGVwIDE6IENoZWNrIFB1bGwgVGFzayBFeGVjdXRpb24gRnJvbSB0aGUgQ2x1c3RlcgoKVGhlIGRpcmVjdCBpbXBhY3Qgb2YgY29ubmVjdGlvbiBmYWlsdXJlIGlzIHB1bGwgdGFzayBleGVjdXRpb24gd2lsbCBub3Qgd29yay4gSWYgU2VydmljZSBBIGNhbiBjb250aW51ZSB0byBwdWxsIGZyb20gU2VydmljZSBCLCB0aGVuIHRoZSBpbmNpZGVudCBjYW4gYmUgZGlzbWlzc2VkIGFzIGZhbHNlIGFsYXJtLCB0aGUgZmVhdHVyZSBvd25lciBjYW4gaW52ZXN0aWdhdGUgZnVydGhlciB0byBzZWUgd2h5IEVjaG8gZmFpbHMuIFRoaXMgY2FuIGJlIHZpc3VhbGl6ZWQgYnkgcHVsbCB0YXNrIGNvdW50IG92ZXIgdGltZSBpbiB0aGUgbGFzdCA4IGhvdXJzIGluIHRoZSBmb2xsb3dpbmcgcXVlcnk6CiAgICBgYGBrdXN0bwogICAgPENvZGUgQmxvY2s+CiAgICBgYGAKCkRpc3JlZ2FyZCB0aGUgbGFzdCBkYXRhIHBvaW50LCBpZiB0aGUgZGF0YSBwb2ludCBpcyBhbHdheXMgYWJvdmUgemVybywgdGhlbiBjb25zaWRlciB0aGUgYWxlcnQgYXMgZmFsc2UgYWxhcm0uIElmIHRoZSBjaGFydCBzb21ldGltZXMgZHJvcHMgdG8gemVybyBvbmUgaG91ciBhZ28gYW5kIHRoZSBudW1iZXIgaXMgbG93IGluIGdlbmVyYWwgKGZvciBpbnN0YW5jZSBsZXNzIHRoYW4gMjApLCBpdCBtZWFucyB0aGUgY3VzdG9tZXIgdHJhZmZpYyBpbiB0aGUgY2x1c3RlciBpcyBsb3cuIEluIHRoaXMgY2FzZSwgb2JzZXJ2ZSBmb3IgYSBsb25nZXIgcGVyaW9kIG9mIHRpbWUuIElmIHRoZSBkYXRhIHBvaW50IGlzIHplcm8gY29uc2lzdGVudGx5IGluIHRoZSBwYXN0IDMwIG1pbnV0ZXMsIHRoZW4gaXQgaXMgYSByZWFsIHByb2JsZW0sIGFuZCBwbGVhc2UgQ2hlY2sgaWYgT3RoZXIgQ2x1c3RlcnMgSW4gdGhlIFJlZ2lvbiBhcmUgSW1wYWN0ZWQuICBPdGhlcndpc2UsIGNvbnRpbnVlIHRvIG9ic2VydmUgc2luY2UgU2VydmljZSBBIGlzIHB1bGxpbmcgU2VydmljZSBCIGp1c3QgZmluZS4=)<br>How to Investigate Service A-To-Service B Connection?<br>### Step 1: Check Pull Task Execution From the Cluster<br>The direct impact of connection failure is pull task execution will not work. If Service A can continue to pull from Service B, then the incident can be dismissed as false alarm, the feature owner can investigate further to see why Echo fails. This can be visualized by pull task count over time in the last 8 hours in the following query:<br>‘‘‘kusto<br><Code Block><br>‘‘‘<br>Disregard the last data point, if the data point is always above zero, then consider the alert as false alarm. If the chart sometimes drops to zero one hour ago and the number is low in general (for instance less than 20), it means the customer traffic in the cluster is low. In this case, observe for a longer period of time. If the data point is zero consistently in the past 30 minutes, then it is a real problem, and please Check if Other Clusters In the Region are Impacted. Otherwise, continue to observe since Service A is pulling Service B just fine. |

### C.2 Mind2web

We show the details about the document generation instruction we use, the different formats of documents, and the examples we generate.

The data includes a step-by-step execution process, each step encompassing HTML code, Tasks, Previous Actions, and the Element and Action of this step.
Note that the Element comes from the HTML code, and if the correct action is not present on the current page, the Element is None, and you should retrieve it from next step’s Previous Actions.

Now your task is to write a comprehensive and adaptable reference document that outlines the general process for completing tasks like the given task. This document should serve as a guide for others to perform similar tasks on the same website in the future. So it should not be limited but can use this data to be the example, and should be general enough.

Please return the complete reference document that adheres to these guidelines.

[User]

The format of the documents should be as follows:
{FORMAT}

The given execution process is as follows:
{EXAMPLE}

We follow Wang et al. () to brainstorm diverse formats of documents for Mind2web dataset used for retrieval, and the results are listed in Table 10.

**Table 10: The description of different formats of documents on Mind2web.**
| Format | Description |
| --- | --- |
| Structured<br>Markdown | - The document must be structured into sections in markdown format.<br>- It should include a task overview, introduction, process steps, and conclusion.<br>- Each step in the process includes detailed explanations for Intent, Prerequisite, HTML Code Reference, Action, Reason, and Result.<br>- The Prerequisite is to specify any conditions or prior actions that must be met or completed before proceeding with the current step in the process.<br>- Ensure that each step is explicitly connected to the next one, and the result is written in the "if-then" schema where the "Intent" of this step is completed, and the outcome "then" is the next step’s Intent.<br>- The HTML Code Reference gives hints of the Action like some ’<button>’, ’<span>’, or other elements or attributes. You need to use the given task as an example.<br>- The Action comes from "Click", "Type", "Hover", "Press Enter". |
| Hierarchical Guideline | - A structured text document with numbered steps for each task.<br>- Each step includes a title, description, the HTML code involved, and the action to be taken.<br>- Previous actions are referenced where necessary, with hyperlinks to the relevant steps.<br>- Appendices for HTML code references, glossary of terms, and FAQs. |
| Tabular<br>Checklist | - A printable checklist with each task and subtask, including checkboxes for completion.<br>- Each checklist item includes a code snippet and the action required.<br>- A troubleshooting section that lists common problems and their solutions.<br>- Tips for what to do when the expected element or action is not available.<br>- References to more detailed instructions or external resources for complex tasks. |
| Narrative<br>Document | An entire description of the execution process without special structures. |

### C.3 Wikihow

I will give you the answer generated by the system in <Generation> and the ground truth answer in <Ground Truth> respectively. Your evaluation will contain five sub-evaluation tasks:

1. Both two answers contain a list of steps. Your task is to extract action items from the provided steps in both answers. The action item is defined as a combination of action and element. Compare the action items to identify similarities. Output the similar action items. Count the count of similar action items.

- Your answer should contain the extracted two action item sets (in the format as a list of strings).
- Your answer should contain a set of similar action items (in the format of a list of strings). Similar action items are those sharing similar intent or achieving similar goals. Each similar action pair in the list should be in the format of "similar action item from action item set1 / similar action item from action item set2"
- Your answer should contain the count of similar action items.

2. Can <Generation> completely solve the user’s question?
- Your answer should be "Yes" or "No".
- Your answer should contain the reason(s) for your choice. You should not focus on the length of the answer or the details of the answer, but you should focus on whether the steps could solve the user’s question and the quality of the steps compared with the ground truth.

Your output should be in the following format in JSON:

[User]

Here is the user’s question <Question>: {Question}
The answer from system <Generation> is: {Generation}
The ground truth answer <Ground Truth> is: {Ground Truth}

## Appendix D Details about RAG System

We take the scenario of Mind2web to show the instructions we use in our RAG-based QA system.

Your response should be in the format of "Answer: C. Action: SELECT Value: Pickup".
The answer is A, B, C…, the Action comes from [CLICK, TYPE, SELECT] and the Value is not always needed.

[User]

<DOC_CONTEXT>: {DOC_CONTEXT}
<CHAT_HISTORY>: {CHAT_HISTORY}
<CURRENT_STEP>: {CURRENT_STEP}

Your response should be in the format of JSON:

[User]

<LOGIC_UNIT>: {LOGIC_UNIT}
<CHAT_HISTORY>: {CHAT_HISTORY}
<CURRENT_STEP>: {CURRENT_STEP}

## Appendix E Limitations

This work focuses on evaluating the effectiveness of Thread by designing how-to questions in three specific scenarios, covering both linear and dynamic how-to questions. However, there are several limitations that point to future directions. First, as our logic-based knowledge base can coexist with the original chunk-based knowledge base, we do not extend our method to factoid questions, such as multi-hop or long-form questions.
Second, the experiments primarily use ChatGPT and LLaMA as the backbones. Future work should include evaluations with other LLMs and alternative retrievers such as Contriever to further validate the effectiveness of Thread.
Lastly, while extracting logic units involves an initial cost in terms of LLM usage, this is a one-time process. Once the knowledge base is constructed, it provides significant advantages for industrial applications, particularly in terms of subsequent updates and maintenance.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 1
questions: My main concern is clarity, as it affects how I understand the presented results. I will be able to give a more accurate evaluation upon clarification.

### Methodology
1. How is the selector in the retrieval pipeline implemented? Does it check pre-requisites by embedding similarity or by string match/ngram overlap?
2. The Linker description says, “Its format varies by LU type, serving as either a query for retrieving other LUs or an entity relationship.” is it like an actual pointer to another LU or is it just some text that you then need to use the retriever to search for other LU? 

### Experiments:
1. How do you verify the reformulation and extraction of documents to make sure there are no hallucinations?
2. For a single task in mind2web there are multiple interactions with the website to accomplish the task. Is this the task setup: given webpage and element choices, (optionally given relevant docs for RAG baselines), choose an element and an action?

### Results
In 5.2 ablation for Retrieval Unit Selector, it says "Comparing chunk selection to LU selection, the performance of Ele. Acc drops by 4.29% and Op.F1 by 3.38%", can you point out which rows on Table 5 is this concluded from? Is it a typo where "Chunk" performance is lower than "Chunk w/o chunk selection"?
soundness: 3
strengths: 1. The authors proposed a novel way to organize document corpus. When compared to the usual chunking methods, it facilitates building logical connections between units, and compared to knowledge-graph approaches, it facilitates building meta connections between units that are larger than the more atomic-level knowledge base entities.
2. This approach is shown to be effective at improving answering how-to questions
summary: 1. The authors proposed to organize documents in Retrieval Augmented Generation systems into Logical Units, each consisting of Prerequisite, Header, Body, Linker, and Metadata. This aims to better answer how-to style questions.
2. To build a Threads knowledge base, LLMs are used to reformulate and extract logical units from a document corpus. To apply this in a RAG system, LU headers are used for embedding search, Prerequisite field is used for filtering, and Linker field is used for finding next steps in a dynamic style how-to questions when there can be multiple execution outcomes.
3. They tested this method on web navigation, wikihow, and human evaluation for incident mitigation, and showed superior performance when compared to other document chunking methods.
weaknesses: ### Clarity
In general the major problem for me is clarity. The paper is a bit hard to follow and requires multiple re-read to understand how the system is supposed to be implemented.

1. For the methodology section, reading the text itself is clearer than trying to parse figure 2 & 3. For figure 2, the arrow from (b) to the right side confused me. Per my understanding, the right panel is an individual view of an LU. Maybe split up the high-level flow with component details into different figures? In figure 3, it was initially hard to match the components to your text description. If you add some formulation and use symbols in both the figure and text, it might make it easier to follow.

2. For the experiment section, it is hard to establish correspondence between components described in the methodology section (sec 3) and their instantiations in the 3 datasets. e.g. I'm confused with how the "execution" part described in sec 3 is implemented in the 3 datasets. 

- Having some symbols/anchors to establish correspondence between the subcomponents and their implementation in experiments might help. 

- Maybe additional columns in table 1 showing design choices for each dataset like: "Dynamic?", "Executable?", "What is being executed" ...

- Before introducing the evaluation metrics, it might be helpful to explain what the task setup is for each dataset, what input is the baseline & your system given, what output is the system expected to produce.

### Some additional results that could be included
1. While the approach seems to perform well compared to existing benchmarks, I'm not really sold on the applicability of this approach. It requires rewrite & extraction of the documents using an LLM before it is even indexed. I would be interested to know how expensive time/cost-wise to implement this, e.g. how it scales in-terms of document length & corpus size etc.
2. The actual embedding search is using only the header field of each LU. It would be interesting to know if indexing using the body help or hurt performance. 
3. Can you still use this organization approach for other 5W questions or would it hinder performance?

---

confidence: 2
contribution: 3
initial_rating: 5
presentation: 3
questions: None
soundness: 2
strengths: 1.	THREAD introduces a novel logic-based organization paradigm that emphasizes the logical flow needed to handle complex how-to questions effectively. 
2.	The paper’s design of logic units with structured components is well-thought-out. Each component serves a specific purpose, enhancing logical continuity and allowing for more coherent, step-by-step responses.
3.	Experimental results show that THREAD outperforms traditional data organization paradigms.
4.	THREAD reduces the number of retrieval units and token length required for generation, optimizing memory usage and computational efficiency.
summary: This paper addresses the limitations of current RAG systems in answering "how-to" questions, which are often complex, multi-step, and dynamic in nature. The authors propose THREAD, a novel data organization paradigm that restructures documents into interconnected "logic units". THREAD enables a more coherent, stepwise approach to answering how-to questions by capturing the logical flow within documents. Experiments demonstrate that THREAD outperforms existing chunk and document-based paradigms.
weaknesses: 1. The experiments mainly compare THREAD with chunk-based and document-based paradigms, with limited discussion on other advanced retrieval techniques, such as graph-based retrieval or hierarchical indexing. 
2. In the industrial setting, the evaluation relies on human engineers. However, the paper does not provide enough details about the evaluation protocols, inter-annotator agreement, or potential biases. 
3. THREAD’s effectiveness appears to depend on the assumption that documents are reasonably structured and easy to parse into logic units. The paper does not discuss THREAD’s robustness when faced with inconsistent or poorly formatted documents, which are common in real-world scenarios.
4. Although THREAD reduces the knowledge base size and required information for generation, the paper lacks a detailed analysis of computational efficiency and scalability when handling very large knowledge bases, potentially limiting its practicality for large-scale applications.
5. The introduction of multiple components and types in each logic unit may increase the complexity of maintaining the knowledge base, especially when documents are updated. While the paper mentions LU updating, it lacks sufficient detail on how updates will be managed and the potential maintenance overhead.
6. The paper does not include a detailed error analysis to identify common failure cases. Understanding these would help clarify THREAD's limitations and guide future improvements.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. The proposed method calls many logic units which act like tools and divide a question to many sub questions or new questions. So The proposed method should be compared with baselines using tool learning, such as Toolllm and ControlLLM, etc. You can find more from "Tool Learning with Large Language Models: A Survey".

2. How to ensure that there are tight connections between LUs? It is easy to form a large number of meaningless nodes. If one LU triggers more than one other LUs, or if a problem triggers more than one LU, how does your approach work? What if none of the Linkers support the answer to the question? 

3. The idea of LU is good, but the construction of LU in practical applications is very troublesome. LU is completely based on human design and is very complex. Do you have any ideas to deal with it? What is the contribution to the performance of each part in LU? The meta data in LU did not described in the Fig.2.

4. The baselines in this paper are not particularly persuasive(some are not based on RAG), and the outcome of LU should be tested in more related baselines.

5. The data sets used in this article are all small-scale data sets. Have you tried large-scale data sets?
soundness: 2
strengths: The paper is written well and points out a good question.
The idea of LU is reasonable.
summary: The paper proposes THREAD, a novel data organization paradigm aimed at enabling current systems to handle how-to questions more effectively. Specifically, we introduce a new knowledge granularity, termed ‘logic unit’, where documents are transformed into more structured and loosely interconnected logic units with large language models. Extensive experiments are conducted across both open-domain and industrial settings.
weaknesses: Objectively speaking, most RAG objects are chunks. This paper does not change this basic background. So the beginning of this article is very misleading. It is recommended not to exaggerate the motivation in the respect of chunks.

The last sentence of contribution # 3 does not seem to be supported by any experiments.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Is the extraction of the Linker component based on a single document or a comprehensive analysis of multiple documents?
2. Can the paper provide a clearer explanation of the unit merging process, including how four parts of two logical units are merged?
3. When performing logical unit extraction, how is content extracted from a document that may contain multiple aspects and themes, especially when the specific question is unknown?
soundness: 4
strengths: Originality:
This paper introduces THREAD, an innovative data organization paradigm that captures the logical structure within documents and the connections between steps through the concepts of logical units and linkers.

Quality:
In terms of experimental design, the paper conducts extensive testing on datasets from both open-domain and industrial environments, with detailed comparisons to existing methods, demonstrating the generalization ability and reliability of its approach. The paper also considers language models of different sizes.

Clarity:
The paper is well-structured, from the problem statement to the methodology, experimental design, and results analysis. The presentation of experimental results is clear, and the use of charts and figures makes the data easy to compare and understand.

Importance:
This paper effectively addresses "how-to" problems in practical applications, demonstrating advantages compared to existing methods.
summary: This paper proposes a novel data organization paradigm named THREAD, aimed at improving the capability of question answering systems in dealing with "how-to" questions, particularly those requiring dynamic, step-by-step solutions. Existing retrieval-augmented generation systems face challenges in connecting documents or chunks when handling such questions. THREAD introduces a new granularity of knowledge known as "logical unit" and adaptively utilizes the Linker to explicitly represent the internal logic between texts. This enables the method to exclude redundant information during the reasoning process and better maintain the coherence of answers. The paper conducts extensive experiments in open-domain and industrial settings, including Web Navigation, Wikipedia Instructions, and Incident Mitigation scenarios. Additionally, the authors provide a detailed analysis of different data organization paradigms, including ICL, SL, RAG based on various chunking approaches and the proposition, etc, to evaluate the superiority of the THREAD paradigm compared to existing methods. This paper addresses a specific and practical problem, proposes a well-motivated approach, and supports its claims through rigorous experimentation.
weaknesses: 1. The description of the extraction and merging process of logical units in the paper is somewhat vague, lacking an in-depth explanation: Although Sections 3.1 and 3.3 of the paper provide introductions, they mainly focus on the functions and roles, with relatively vague descriptions of specific implementation strategies, especially for the crucial Linker component. Additionally, the specific implementation process of unit merging is not clear, such as how to merge the various parts of two logical units.
2. The THREAD proposed in the paper represents a novel paradigm specifically tailored for addressing "how-to" questions, yet it poses significant implementation challenges in other real-world scenarios, particularly in establishing Linkers between logical units, and exhibits limited method scalability. The paper primarily focuses on how to better extract logical units, while the precise recognition and extraction of relationships between any two text segments or documents are not discussed in detail.

Meta-review: This paper presents a way to segment sources in RAG systems and focuses on a particular type of questions and show strong results in end to end experiments.  The paper has a lot of experiments and the motivation is clear.  That said, I am not particularly excited about the paper due to a few reasons:

1.  Focus on a relatively narrow type of questions.
2.  A fairly specific way to segment documents (which the second reviewer points out as well) and I am uncertain whether this will generalize to a variety of sources.

Given this I think this paper will be more well suited as a tech report and not a paper at ICLR.  Most reviewers also think similarly and have a lukewarm response to the paper.

Decision: Reject