Venue: NAACL 2024

# Tree-of-Question: Structured Retrieval Framework

for Korean Question Answering Systems

 Dongyub Lee1, Younghun Jeong1, Hwayeon Kim1, Hongyeon Yu1, Seunghyun Han1,

Taesun Whang1, Seungwoo Cho1, Chanhee Lee1, Gunsu Lee1, Youngbum Kim1\({}^{\dagger}\)

###### Abstract

We introduce Korean language-specific RAG-based QA systems, primarily through the innovative Tree-of-Question (ToQ) methodology and enhanced query generation techniques. We address the complex, multi-hop nature of real-world questions by effectively integrating advanced LLMs with nuanced query planning. Our comprehensive evaluations, including a newly created Korean multi-hop QA dataset, demonstrate our method's ability to elevate response validity and accuracy, especially in deeper levels of reasoning. This paper not only showcases significant progress in handling the intricacies of Korean linguistic structures but also sets a new standard in the development of context-aware and linguistically sophisticated QA systems.

+
Footnote †: [https://clouda.ai/hyperclova](https://clouda.ai/hyperclova)

+
Footnote †: [https://clouda.ai/hyperclova](https://clouda.ai/hyperclova)

+
Footnote †: [https://clouda.ai/hyperclova](https://clouda.ai/hyperclova)

## 1 Introduction

Recent advancements in Large Language Models (LLMs) have revolutionized information seeking and text generation, with real-world applications such as Bing Chat1, Perplexity.ai2, and Google Bard3 leading the way. These systems utilize the Retrieval Augmented Generation (RAG) methodology, where responses are crafted based on information extracted from retrieved documents. This approach allows these platforms to provide answers that not only are contextually relevant but also cite the sources they reference, enhancing the reliability and transparency of the information provided. This strategy is particularly effective in addressing the inherent issue of hallucinations in LLMs, where LLMs might generate plausible but factually incorrect information.

Footnote 1: [https://www.bing.com/chat](https://www.bing.com/chat)

Footnote 2: [https://www.perplexity.ai](https://www.perplexity.ai)

Meanwhile, a system named "Naver Cue:"4 [Yu et al., 2023] has emerged as a noteworthy addition to the realm of question and answering systems, specifically tailored to the Korean language. It is a system grounded in HyperCLOVAX5[Kim et al., 2021; Shin et al., 2022; Yoo et al., 2024], a Korean Large Language Model with billions of parameters. Utilizing the RAG methodology and leveraging Naver's search engine, "Naver Cue:" efficiently provides contextually relevant and accurate responses. The integration of HyperCLOVAX into this system enables a sophisticated understanding and processing of Korean language nuances, ensuring precise and relevant answers. This makes "Naver Cue:" a notable advancement in the domain of LLM-based QA systems tailored for Korean language.

Footnote 6: [https://dav.gonek.com](https://dav.gonek.com)

In the realm of RAG-based QA systems powered by LLMs, the operational mechanism typically unfolds in a structured three-step process. Initially, the system generates a query derived from the user's question, effectively translating the user's inquiry into a format suitable for search engines. Following this, the query is processed through the search engine, which conducts a comprehensive search to gather relevant information. The final step involves generating a response by synthesizing and summarizing the search results into a coherent answer.

A critical aspect of this process is the creation of queries that are specifically optimized for each search engine. This optimization is crucial as it directly influences the relevance and accuracy of the information retrieved. Furthermore, the nature of questions posed by users in such systems is predominantly multi-hop. These multi-hop questions are multifaceted, often characterized by their inherent ambiguities or the need to collate information from multiple sources [Mavi et al., 2022; Amplayo et al., 2022; Trivedi et al., 2022]. This complexity poses additional challenges in query formulation, making it imperative for the

to have advanced understanding and processing capabilities.

Particularly in the case of the Korean language, accurately interpreting the user's question becomes more challenging due to the duality and connotation of words Park et al. (2020); Kim et al. (2021). Korean often involves subtle nuances and implied meanings that can significantly alter the context of a query. Additionally, Korean is an agglutinative language, characterized by its unique grammatical structure where particles follow nouns, and the stems of verbs or adjectives are followed by endings. These endings express various grammatical properties, adding layers of complexity to the language Lee et al. (2020); Yang (2021); Son et al. (2022). These linguistic features can lead to difficulties in generating search queries that precisely mirror the user's intent. Therefore, RAG-based QA systems designed for Korean must possess enhanced capabilities to discern and reflect these subtleties.

Recent research in closed-book QA systems utilizes metrics like ROUGE-L and Disambig-F1 for performance evaluation, comparing model predictions with ground truth answers Lin (2004); Amplayo et al. (2022). These metrics assess end-to-end performance, highlighting how closely model responses match expected answers. However, they fall short in evaluating specific aspects crucial to LLM-based QA systems such as query generation, document retrieval, and response generation. In real-world scenarios, where correct answers aren't predetermined, this becomes a challenge. To address these limitations, new metrics like citation recall/precision Gao et al. (2023) and FactScore Min et al. (2023) have been introduced. These focus on evaluating the system's ability to reference relevant documents and the relevance of summarized responses to posed questions.

Despite these advancements, a notable gap remains: there is currently no established metric for evaluating the appropriateness of the queries generated by the in-house search engine in response to the user's questions. This highlights a crucial area for further research and development, as the ability to generate accurate and relevant queries is fundamental to the success of real-world QA systems.

To effectively address the challenges in the current landscape of LLM-based Open-domain QA systems, particularly for the Korean language, our research introduces several pivotal contributions, summarized as follows:

* **Enhanced Query Generator in Korean RAG-based Long-form QA Systems:** We propose an advanced role for the Query Planner, optimizing queries from user inquiries for search engine compatibility. This aims to improve the accuracy and relevance for Korean language nuances.
* **Tree-of-Question for Multi-Hop Reasoning:** Introducing a structured Tree-of-Question concept, our approach enhances the system's capacity to process multi-hop questions in Korean.
* **Novel Evaluation Method for Query Planner:** We develop a new method for evaluating the Query Planner in multi-hop query processing systems, utilizing LLMs for both offline and online assessments.

## 2 Task Definition

In LLM-based Retrieval Augmented Generation (RAG) QA systems, we define the process as a sequence of functions, each transforming an input to produce an output that serves as the input for the subsequent function. Let \(q\) be the user's question. Then, the process can be formalized using the following notation:

1. _Query Generation_: A function \(f_{QG}\) takes \(q\) and generates a query \(Q\).

\[Q=f_{QG}(q)\]

This involves interpreting \(q\) and restructuring it into a format optimized for the in-house search engine.

2. _Document Retrieval_: A function \(f_{DR}\) takes \(Q\) and retrieves a set of documents \(D\) from the in-house search engine.

\[D=f_{DR}(Q)\]

These documents are relevant to the query and contain information pertinent to answering \(q\).

3. _Response Generation_: A function \(f_{RG}\) takes \(q,~{}Q\), and \(D\), and generates a final response \(R\).

\[R=f_{RG}(q,\,Q,D)\]

This involves synthesizing information from \(D\) in the context of \(q\) and \(Q\) to provide a comprehensive and accurate answer to the user's question.

In our research, we emphasize the development of an effective _Query Generator_ within the LLM-based RAG QA framework. We operate under the 

assumption that the in-house search engine and the _Response Generator_ are already established and functional. Our focus is primarily on enhancing the _Query Generator_, \(f_{QG}\), which is responsible for transforming user questions \(q\) into optimized queries \(Q\).

## 3 Enhanced Query Planning with Tree-of-Question and Query Evaluator

In the Korean RAG-based long-form QA system, complex questions often require structuring into simpler, searchable queries. Previous studies have explored various aspects of multi-hop reasoning in closed-book QA systems. In the study by Min et al. (2019), question types requiring multi-hop reasoning were classified into three primary categories: bridging, intersection, and comparison. Furthermore, the research conducted by Amplayo et al. (2022), which draws inspiration from reasoning chains in LLMs. This approach entails formulating explanations as a sequence of interconnects. Additionally, Trivedi et al. (2022); Press et al. (2022) proposed methods of generating questions sequentially and creating follow-up questions when necessary. This sequential approach is valuable for developing a deeper understanding of the topic in question and ensuring comprehensive coverage.

However, these methods encounter limitations in responding to questions that search for multiple queries in parallel and then synthesize these answers. Such complex scenarios, referred to as the "Hybrid" type in Table 1, require a more nuanced approach that combines elements of different reasoning types. This gap highlights the need for advanced methodologies capable of handling these hybrid multi-hop reasoning challenges effectively.

### Tree of Questions

As illustrated in Figure 1, we propose a novel method, Tree-of-Question (ToQ), to decompose and structure complex queries in a tree-like format. This approach is inspired by the multi-hop question concept in QA and the Tree-of-Thought (ToT) methodology from recent advancements in LLMs Yao et al. (2023). Our system logically connects and structures questions to facilitate planned retrieval and comprehensive search processes in multi-hop reasoning scenarios.

Algorithm 1 illustrates the process of the ToQ method. The Root node represents the user's original question. Each node in the tree corresponds to a sub-question derived from or related to the original query or its preceding nodes. The level of a node indicates its depth in the tree, representing the sequential steps needed to reach a conclusive answer. The ToQ dynamically expands as it decomposes complex questions into simpler, interconnected sub-questions. When a dependency between Nodes is identified, especially in cases similar to Bridging or Hybrid scenarios, as illustrated in Table 1, the Answer Integrator is utilized to fill in the necessary answers in the [ANS] portion of the question. Finally, the ToQ process terminates when the original user's question can be satisfactorily answered using the responses from the created nodes, a determination made using the Query Evaluator.

```
1:\(Root\): The original question \((Q)\)
2:\(Level\): The depth in the tree, representing the number of nodes to reach an answer
3:
4:functionTreeOfQuestion(Node)
5: Determine \(Level\) of \(Node\) and increment hop count
6: Identify dependent \(Nodes\), use Answer Integrator if related to parent \(Node\) and modify \(Root\) or \(Node\) if needed
7: Generate query from the (modified) \(Root\) or \(Node\)
8: Generate response based on the query and retrieved documents
9:\(Eval\leftarrow\) Evaluate query and response using Query Evaluator
10:if\(Eval\) is positive then
11:return
12:endif
13:\(DecomposedNodes\gets\) Decompose \(Node\) into sub-questions if needed
14:for each \(SubNode\) in \(DecomposedNodes\)do
15: Create new \(Node\) for \(SubNode\)
16: Update \(Level\) for new \(Node\)
17: Recursively call TreeOfQuestion(\(SubNode\))
18:endfor
19:endfunction
```

**Algorithm 1** Tree of Questions 

questions into optimized queries for retrieving relevant documents from an in-house search engine. Given \(k\) in-context exemplars of question-query pairs \([(q_{1},Q_{1}),\ldots,(q_{k},Q_{k})]\), along with an instruction, the query generator generates a query \(Q\) for a question \(q\). This process is pivotal in accurately sourcing information, ensuring that the generated queries \(Q\) are precisely formulated to

Figure 1: Architecture of Tree of Questions.

 

align with the user's inquiry \(q\), as demonstrated in Appendix Prompt B.

### Query Evaluator

The query evaluator, as utilized in line 9 of Algorithm 1, plays a crucial role in determining if the original question, denoted as \(q\), is adequately addressed by the generated queries \(Q\) and responses. It uses a LLM to evaluate these elements on four key aspects. **Semantic Coherence** assesses the logical flow and relevance of the response to \(q\), scored from 1 (no coherence) to 10 (perfect coherence). **Answerability** measures the likelihood of the response directly addressing \(q\), with a confidence level expressed as a percentage from 0% to 100%. Each response's **Overall Assessment Score** is computed by averaging the Semantic Coherence score and the Answerability score (after converting it from a percentage to a 1-10 scale). The evaluator also provides a **Response Validity** indicator, a binary (true/false) metric that determines the adequacy of responses in answering \(q\) based on coherence and answerability assessments. This indicator is crucial in determining the applicability of the responses to real-world questions, providing a clear binary decision. Details about the evaluator's instruction prompt can be found in Appendix Prompt B.

## 4 Experiments

### Dataset Collection

Existing English datasets for multi-hop QA, such as HotpotQA Yang et al. (2018), 2WikiMulti-hopQA Ho et al. (2020), and ASQA Stelmakh et al. (2022), provide a foundation for evaluating multi-hop QA on English benchmarks. These datasets consist of questions and their corresponding answers in a closed-book setting, focusing on generating accurate answers to given questions and documents. To address the absence of a Korean dataset suitable for multi-hop QA, we have taken the initiative to create a dataset specifically tailored for evaluating Korean multi-hop QA. We have meticulously crafted 200 questions that require multi-reasoning capabilities based on the types described in Table 1. These questions are human-generated to specifically address diverse aspects of multi-reasoning, ensuring a comprehensive evaluation of our approach. In creating these questions, we strictly adhered to ethical guidelines and carefully recognized any sensitive information, ensuring the content was appropriate and non-sensitive. Our goal is to release this rigorously curated subset to the public, contributing a valuable resource to the field of Korean multi-hop QA and encouraging further research with practical and applicable evaluation tools. To further demonstrate the robustness of our method, we additionally extracted a dataset of 1,000 questions from the "Naver Cue:" real-world Korean QA system logs. In this process, we meticulously anonymized the data to not only uphold privacy standards but also to comply with privacy regulations and ethical standards. This careful approach ensures the protection of user privacy while allowing us to validate our method effectively in a real-world context.

### Evaluation Metrics

Metrics such as ROUGE and Disambig-F1, traditionally used in closed-book QA systems for comparing model-predicted answers against groundtruth data Lin (2004); Amplayo et al. (2022), are well-suited for end-to-end evaluation where definitive answers exist. However, It is important to note that in real-world applications, the performance of the query planning module cannot be accurate as the correct answers to user queries are often unknown or variable, posing a significant challenge in assessing the module's effectiveness in practical scenarios. Therefore, our focus is on evaluating the query planning process itself, for which we employ the automated metrics proposed in Section 3.4. Additionally, to ensure the reliability of the Query Evaluator, we conduct a human evaluation as described in Section 5.1. This methodological integration ensures a robust and comprehensive assessment of the query planning component.

### Baselines

We compare our proposed method with several established baselines, each representing a unique approach to RAG-based QA systems. Direct GenerationIn this approach, a query generation model directly produces a single query from the question, which is then used for document retrieval. This method focuses on achieving results through a single-time retrieval process based on the initial query. Chain-of-ThoughtThis method involves the model first generating a Chain-of-Thought (CoT) 

in response to a question before delivering the final answer. It represents a thoughtful, step-wise approach to query generation and information retrieval, as detailed in various works (Wei et al., 2022; Yoran et al., 2023; Liu et al., 2023).

Previous ContextBuilt on the CoT method, 'Previous Context' method follows a multi-step retrieval approach. It triggers retrieval using the previous context as the query. This method, including works like IRCoT (Trivedi et al., 2022), emphasizes the use of ongoing context for progressive information retrieval.

Self-AskAn extension of the CoT prompting, 'Self-Ask' method differs by having the model explicitly formulate the next follow-up question it intends to answer. It uses a search engine to respond to these sub-questions instead of relying solely on the language model. This method is explored in Jiang et al. (2023).

### Main Results

Comparison with BaselinesOur evaluation begins with a focused analysis on a subset of 200 questions specifically requiring multi-hop reasoning, as illustrated in Table 2. In this targeted evaluation, the Tree of Questions (ToQ) method significantly outperforms established baselines, achieving a Response Validity of 74.0% and demonstrating strong scores in Semantic Coherence and Answerability at 7.02 and 60.45, respectively. This superior performance in a complex multi-hop reasoning context underscores the effectiveness of the ToQ framework in handling intricate queries.

Following the targeted analysis on multi-hop reasoning, we extend our evaluation to a broader dataset of 1,000 randomly sampled questions, the performance of which is detailed in Table 3. This comprehensive evaluation demonstrates that the ToQ method consistently maintains its high level of performance across a diverse range of question types and complexities. The ToQ framework's robust and adaptable performance across a wide range of QA scenarios, including both focused multi-hop reasoning and a diverse set of questions, highlights its versatility and reliability. Its consistent efficacy demonstrates the method's ability to accurately address questions of varying complexity and depth.

Performance Analysis by Tree DepthWe evaluate the performance of our method by examining the Response Validity at various depths within the tree. Table 4 presents the current performance of response validity at each tree level. The tree is limited to a maximum depth of four levels, focusing on the effectiveness of our approach in decomposing and addressing complex queries. This analysis provides insight into how the depth of reasoning impacts the quality of responses generated by our system.

As we delve deeper into the tree levels, we observe an increase in Response Validity. This improvement can be attributed to the increased specificity and context-awareness in sub-questions at deeper levels, and the more focused information retrieval that accompanies this specificity.

## 5 Analysis

### Correlation of Query Evaluator with Human Judgment

We focus on assessing the accuracy of the Query Evaluator by comparing its evaluations with those made by human annotators. The goal is to establish the degree of correlation between automated

\begin{table}
\begin{tabular}{l c c c c} \hline \hline Method & S. Coh & Ans. & O. Ass & R. Val (\%) \\ \hline \multicolumn{5}{c}{Single-time Retrieval} \\ \hline Direct Generation & 6.78 & 50.15 & 5.46 & 58.5 \\ Chain-of-Thought & 6.83 & 54.75 & 5.69 & 64.0 \\ \hline \multicolumn{5}{c}{Multi-time Retrieval} \\ \hline Previous Context & 7.02 & 57.07 & 6.11 & 66.0 \\ Self-ask & 7.01 & 59.25 & 6.16 & 69.5 \\ ToQ (ours) & **7.02** & **60.45** & **6.18** & **74.0** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance comparison of baseline methods on the dataset of 200 questions requiring multi-hop reasoning. Abbreviations: S. Coh (Semantic Coherence), Ans. (Answerability), O. Ass (Overall Assessment), R. Val (Response Validity). The methods are categorized into Single-time and Multi-time Retrieval.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline Method & S. Coh & Ans. & O. Ass & R. Val (\%) \\ \hline \multicolumn{5}{c}{Single-time Retrieval} \\ \hline Direct Generation & 6.95 & 64.94 & 6.24 & 83.4 \\ Chain-of-Thought & 6.95 & 66.79 & 6.33 & 86.7 \\ \hline \multicolumn{5}{c}{Multi-time Retrieval} \\ \hline Previous Context & 7.09 & 68.78 & 6.57 & 86.9 \\ Self-ask & 7.09 & 69.30 & 6.57 & 88.0 \\ ToQ (ours) & **7.12** & **69.33** & **6.62** & **89.0** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Performance comparison of baseline methods on the dataset of random sampled 1,000 questions.

 

and human assessments, thereby validating the reliability and credibility of the Query Evaluator's performance in real-world scenarios.

First, to validate our automated system, we use the Inter-Annotator Agreement (IAA) to measure consistency among human annotators. As noted in Appendix C, the high IAA scores indicate a significant agreement, confirming the reliability of our human judgment benchmark. Second, our analysis extends to examining the correlation between each metric component used by the Query Evaluator (such as Semantic Coherence, etc.) and human annotations. The detailed findings, presented in Appendix D, include Pearson's correlation coefficients for each metric. These coefficients, reveal how closely each aspect of the Query Evaluator's assessment aligns with human judgment.

### Improvement in Handling Failures with Tree of Questions

Analyzing the transition from single-time retrieval failures to ToQ success, we observe a significant improvement. Out of 83 failures in single-time retrieval at level-1, 31 questions (37.3%) are successfully addressed using the ToQ approach, with increasing success rates at deeper levels.

### Qualitative Analysis

We present a qualitative analysis of our ToQ method, focusing on its ability to effectively handle complex queries, as illustrated in Appendix Figure 3. For instance, in the case of the single-time retrieval method applied to the question, _"Recommend a deposit that is advantageous to young people born in 1996. Please tell me that there are no restrictions on the family's wealth,"_ the method exhibits limitations in adequately addressing the query's nuances. In contrast, our ToQ method constructs a question tree node corresponding to a bridging case with an additional depth of two levels. This enables the ToQ to generate more appropriate queries for searching, ultimately providing a more accurate and relevant answer to the original question. The qualitative comparison underscores the enhanced capability of the ToQ method in handling complex, multi-faceted questions.

### Error Case Studies

In our analysis, we identify several types of error cases that pose challenges to our Tree of Questions method. These cases shed light on areas where further improvement is needed.

Inability to Decompose QuestionsSome questions, such as _"Please show me a photo of the Go-chon area in 1977,"_ cannot be effectively decomposed into simpler queries, leading to a failure in the ToQ process. These types of questions, which are inherently complex and lack a straightforward decomposition path, comprise approximately 10% of the questions in our dataset, indicating a significant area for potential improvement in handling such intricate questions.

Long-tail QuestionEven with a successfully generated query, the absence of reliable documents on the search engine can lead to errors. This is common in long-tail questions such as hyper-specific legal questions, inquiries into particular cultural practices, or detailed comparisons of obscure products.

## 6 Conclusion

In this paper, we introduce advancements in RAG-based QA systems for Korean, focusing on the Tree-of-Question (ToQ) methodology and enhanced query planning. Our evaluations show the ToQ method's effectiveness in multi-hop reasoning and its adaptability across a comprehensive dataset. Notably, ToQ significantly improves handling complex Korean language queries by enabling deeper reasoning. Additionally, we present a novel evaluation method in a detailed Korean multi-hop QA dataset. Our contributions pave the way for more accurate and context-sensitive QA systems, especially for languages with unique challenges like Korean.

\begin{table}
\begin{tabular}{c c} \hline \hline Tree Depth & Response Validity (\%) \\ \hline Level 1 & 58.5 \\ Level 2 & 68.5 \\ Level 3 & 72.0 \\ Level 4 & **74.0** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance of Response Validity at different levels of the Tree of Questions, showing a clear trend of increasing satisfaction rate with deeper levels.

\begin{table}
\begin{tabular}{l c c} \hline \hline
**ToQ Level** & **Resolved Cases** & **Rate (\%)** \\ \hline Level-1 (Initial Failure) & 0 & 0.0 \\ Level-2 & 20 & 24.1 \\ Level-3 & 27 & 32.5 \\ Level-4 & 31 & 37.3 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Resolution rates of single-time retrieval failures at different levels of the Tree of Questions.

 



## Limitations

Language Scope and Future ExpansionWhile our study offers significant insights into multi-hop question answering for the Korean language, leveraging a model specifically designed for Korean, it's important to recognize its limitations in terms of language scope. Our experiments were conducted exclusively on Korean datasets, validating the effectiveness of our method in this specific linguistic context. However, to broaden the applicability and validate the universality of our approach, we plan to extend our experiments to English datasets. This expansion will involve using other Large Language Models as the backbone.

Challenges in Addressing Long-tail QuestionsAnother limitation in our approach arises when dealing with long-tail questions. These questions often pertain to highly specialized or niche topics, such as detailed legal inquiries, specific cultural practices, or comparisons of obscure products. Even if our system successfully generates a query for such questions, the limitation lies in the availability of relevant and reliable documents within the search engine's database. The scarcity of comprehensive information on these niche topics can result in inaccuracies or incomplete answers.

## References

* K. Amplayo, K. Webster, M. Collins, D. Das, and S. Narayan (2022)Query refinement prompts for closed-book long-form question answering. arXiv preprint arXiv:2210.17525. Cited by: SS3.4.
* S. Borgeadu, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driessche, J. Lespiau, B. Damoc, A. Clark, et al. (2022)Improving language models by retrieving from trilions of tokens. In International conference on machine learning, pp. 2206-2240. Cited by: SS3.4.
* T. Gao, H. Yen, J. Yu, and D. Chen (2023)Enabling large language models to generate text with citations. External Links: Link Cited by: SS3.4.
* X. Ho, A. Duong Nguyen, S. Sugawara, and A. Aizawa (2020)Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01060. Cited by: SS3.4.
* J. Huang, X. Chen, S. Mishra, H. Steen Zheng, A. Wei Yu, X. Song, and D. Zhou (2023)Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798. Cited by: SS3.4.
* Z. Jiang, F. F. Xu, L. Gao, Z. Sun, Q. Liu, J. Dwivedi-Yu, Y. Yang, J. Callan, and G. Neubig (2023)Active retrieval augmented generation. arXiv preprint arXiv:2305.06983. Cited by: SS3.4.
* M. Khalifa, L. Logeswaran, M. Lee, H. Lee, and L. Wang (2023)Fewshot reranking for multi-hop qa via language model prompting. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vol. , pp. 15882-15897. Cited by: SS3.4.
* U. Khandelwal, O. Levy, D. Jurafsky, L. Zettlemoyer, and M. Lewis (2019)Generalization through memorization: nearest neighbor language models. arXiv preprint arXiv:1911.00172. Cited by: SS3.4.
* O. Khattab, K. Santhanam, X. Lisa Li, D. Hall, P. Liang, C. Potts, and M. Zaharia (2022)Demonstrate-search-predict: composing retrieval and language models for knowledge-intensive nlp. arXiv preprint arXiv:2212.14024. Cited by: SS3.4.
* T. Khot, H. Trivedi, M. Finlayson, Y. Fu, K. Richardson, P. Clark, and A. Sabharwal (2022)Decomposed prompting: a modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406. Cited by: SS3.4.
* B. Kim, H. Kim, S. Lee, G. Lee, D. Kwak, D. Heyon Jeon, S. Park, S. Kim, S. Kim, D. Seo, et al. (2021)What changes can large-scale language models bring? intensive study on hyperclova: billions-scale korean generative pretrained transformers.. arXiv preprint arXiv:2109.04650. Cited by: SS3.4.
* A. Lazaridou, E. Gribovskaya, W. Stockowiec, and N. Grigorev (2022)Internet-augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115. Cited by: SS3.4.
* D. Lee, M. Shin, I. Whang, S. Cho, B. Ko, D. Lee, E. Kim, and J. Jo (2020)Reference and document aware semantic evaluation methods for korean language summarization. arXiv preprint arXiv:2005.03510. Cited by: SS3.4.
* C. Lin (2004)Rouge: a package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81. Cited by: SS3.4.
* Xiangyang Liu, Tianqi Pang, and C. Fan (2023)Federated prompting and chain-of-thought reasoning for improving llmns answering. arXiv preprint arXiv:2304.13911. Cited by: SS3.4.
* A. Mallen, A. Asai, V. Zhong, R. Das, D. Khashabi, and H. Hajishirzi (2023)When not to trust language models: investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of

the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 9802-9822.
* Mavi et al. [2022] Vaibhav Mavi, Anubhav Jangra, and Adam Jatowt. 2022. A survey on multi-hop question answering and generation. _arXiv preprint arXiv:2204.09140_.
* Min et al. [2023] Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. _arXiv preprint arXiv:2305.14251_.
* Min et al. [2019] Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2019. Multi-hop reading comprehension through question decomposition and rescoring. _arXiv preprint arXiv:1906.02916_.
* Nakano et al. [2021] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: browser-assisted question-answering with human feedback (2021). _URL https://arxiv. org/abs/2112.09332_.
* Park et al. [2020] Kyubyong Park, Joohong Lee, Seongbo Jang, and Davoon Jung. 2020. An empirical study of tokenization strategies for various korean nlp tasks. _arXiv preprint arXiv:2010.02534_.
* Press et al. [2022] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. _arXiv preprint arXiv:2210.03350_.
* Qin et al. [2023] Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlu Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang, et al. 2023. Webcpm: Interactive web search for chinese long-form question answering. _arXiv preprint arXiv:2305.06849_.
* Ram et al. [2023] Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. _arXiv preprint arXiv:2302.00083_.
* Shin et al. [2022] Seongjin Shin, Sang-Woo Lee, Hwijeen Ahn, Sungdong Kim, HyoungSeok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, Woomyoung Park, Jung-Woo Ha, et al. 2022. On the effect of pretraining corpora on in-context learning by a large-scale language model. _arXiv preprint arXiv:2204.13509_.
* Son et al. [2022] Suhyune Son, Chanjun Park, Jungseob Lee, Midan Shim, Chanhee Lee, Kinam Park, and Heiseok Lim. 2022. Korean and multilingual language models study for cross-lingual post-training (xpt). _Journal of the Korea Convergence Society_, 13(3):77-89.
* Stelmakh et al. [2022] Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. 2022. ASQA: Factoid questions meet long-form answers. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 8273-8288, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Sun et al. [2022] Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. 2022. Recitation-augmented language models. _arXiv preprint arXiv:2210.01296_.
* Trivedi et al. [2022] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. _arXiv preprint arXiv:2212.10509_.
* Wei et al. [2022] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. _Advances in Neural Information Processing Systems_, 35:24824-24837.
* Yang [2021] Kichang Yang. 2021. Transformer-based korean pretrained language models: A survey on three years of progress. _arXiv preprint arXiv:2112.03014_.
* Yang et al. [2018] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. _arXiv preprint arXiv:1809.09600_.
* Yao et al. [2023] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. _arXiv preprint arXiv:2305.10601_.
* Yao et al. [2022] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. _arXiv preprint arXiv:2210.03629_.
* Yoo et al. [2024] Kang Min Yoo, Jaegeun Han, Sookoyo In, Heewon Jeon, Jisu Jeong, Jaewook Kang, and et al. Hyunwook Kim. 2024. Hyperclova x technical report. _arXiv preprint arXiv:2404.01954_.
* Yoran et al. [2023] Ori Yoran, Tomer Wolfson, Ben Bogin, Uri Katz, Daniel Deutch, and Jonathan Berant. 2023. Answering questions by meta-reasoning over multiple chains of thought. _arXiv preprint arXiv:2304.13007_.
* Yu et al. [2023] Hongyeon Yu, Seung Hak Yu, and Young Bum Kim. 2023. Naver cue: Search service based on large language models. _Communications of the Korean Institute of Information Scientists and Engineers_, 41:34-41.
* Yu et al. [2022] Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2022. Generate rather than retrieve: Large language models are strong context generators. _arXiv preprint arXiv:2209.10063_.

 

Algorithm of Tree of Questions

## Appendix B Prompt Examples

[frame=single,leftmargin=*] 

* [frame=single,leftmargin=*] 

* [frame=single,leftmargin=*] 
\begin{tabular}{}

\end{table}
Table 1: Prompt BL Query Operator 

Inter-Annotator Agreement (IAA) Measurements of Query Evaluator

In this section, we present an in-depth analysis of the Inter-Annotator Agreement (IAA) for our Query Evaluator. The IAA is a crucial metric in evaluating the consistency and reliability of human annotators when assessing the outputs generated by our Query Evaluator. It serves as an indicator of the degree to which different annotators provide similar ratings, thereby offering insights into the validity and interpretability of the Query Evaluator's performance.

To conduct this analysis, we engaged five human annotators, authors of this paper, to assess a sample of 100 queries processed by the Query Evaluator. The queries were evaluated based on predefined criteria, with the aim to compare the consistency of the human annotators' judgments. Two distinct IAA (Inter-Annotator Agreement) measurements were employed: the Direct Generation IAA and the Tree-of-Question (TQO) IAA.

As illustrated in the Table 6, both IAA measurements exhibit substantial levels of agreement among the annotators. In the Direct Generation IAA, the Proportional Agreement (PA) was noted as 0.872, indicating a high level of consensus among annotators in their evaluations. Similarly, the Fleiss' Kappa value of 0.744 in this measurement suggests a substantial agreement beyond chance.

In the ToQ retrieval IAA, there was a slight increase in PA to 0.892, indicating an even higher level of agreement among the annotators for this set of queries. The Fleiss' Kappa value of 0.738, although slightly lower than in the Direct Generation scenario, still indicates a substantial agreement level.

The Probability of Chance Agreement (PE) in both measurements also reflects noteworthy observations. For the Direct Generation IAA, the PE is 0.500288, while for the ToQ retrieval IAA, it is higher at 0.5882. These values indicate that while there is some element of chance agreement, the high Fleiss' Kappa values demonstrate that the majority of the agreement among annotators is due to their consistent judgment rather than chance.

The consistency in these IAA measurements is a testament to the reliability of human annotators in evaluating the queries processed by the Query Evaluator. This consistency also affirms the robustness of the Query Evaluator's output, as it aligns closely with human judgment, which is critical in ensuring the practical applicability of the Query Evaluator in real-world scenarios.

## Appendix D Correlation Analysis Between Human and Query Evaluator Metrics

In this section, we present the results of our correlation analysis between the consensus annotations from five annotators and the metrics computed by our Query Evaluator. We employ a majority voting system to aggregate the binary (True/False) annotations for each query, resulting in a representative consensus for each. Subsequently, we calculate both Pearson and Spearman correlation coefficients to understand the linear and monotonic relationships, respectively, between these consensus annotations and each metric of the Query Evaluator.

Majority Voting AggregationTo aggregate the annotations, we implement a majority voting mechanism. For each query, we determine the most common annotation (True or False) among the five annotators. This approach allows us to capture the dominant trend in human judgment for each query.

Correlation Coefficient CalculationWe calculate the Pearson and Spearman correlation coefficients for the following metrics of the Query Evaluator against the aggregated annotations: 1) Semantic Coherence, 2) Answerability, 3) Overall Assessment Score, and 4) Response Validity.

Each metric is correlated with the consensus annotation to gauge its alignment with human judgment. Pearson correlation was used to assess the linear relationship, while Spearman correlation was employed to understand the rank-order relationship.

Each metric is correlated with the consensus annotation to gauge its alignment with human judgment. Pearson correlation is used to assess the linear relationship, while Spearman correlation is employed to understand the rank-order relationship.

Results of Pearson Correlation AnalysisAs described in Figure 2, pearson correlation analysis yields the following results:

\begin{table}
\begin{tabular}{l c c c} \hline
**Measurement** & **PA** & **PE** & **Fleiss’ Kappa** \\ \hline Direct Generation & 0.872 & 0.500 & 0.744 \\ Tree-of-Question & 0.892 & 0.588 & 0.738 \\ \hline \end{tabular}
\end{table}
Table 6: Inter-Annotator Agreement (IAA) Measurements.

 

* Semantic Coherence: For the 'Direct Generation' method, the Pearson correlation coefficient is 0.3804, indicating a moderate positive relationship with human annotations. For the 'Tree-of-Question' method, the correlation is lower at 0.2128. The lower correlation of Semantic Coherence compared to other metrics can be attributed to the fact that it tends to achieve some level of coherence by mentioning content related to the user's question, even if the question isn't answered directly. This distribution of scores ranging from 6 to 7 points suggests that the metric may not effectively capture the depth or relevance of the answer to the user's query, as it may assign relatively high scores even when the answer is not fully satisfying in terms of providing a direct response.

Figure 3: Qualitative example of Tree-of-Questions Framework.

Figure 2: Comparison of Pearson Correlation Coefficients for ‘Direct Generation’ and ‘Tree of Questions’ methods, illustrating the distinct performance characteristics of each in terms of Semantic Coherence, Answerability, Overall Assessment, and Response Validity.

 

* Answerability: The 'Direct Generation' method shows a strong positive correlation of 0.7757, suggesting high agreement with human judgment. The 'Tree-of-Question' method has a correlation of 0.7061.
* Overall Assessment Score: This metric also demonstrates a strong positive correlation for both methods, with 'Direct Generation' at 0.7593 and 'Tree-of-Question' at 0.6550.
* Response Validity: The strongest correlation with human annotations is observed in the 'Response Validity' metric, with 'Direct Generation' at 0.7764 and 'Tree-of-Question' at 0.6686.

These results indicate the overall assessment and response validity are particularly strong indicators of human judgment across both methods.

## Appendix E Experimental Setup

As outlined in Section 2, our experimental framework assumes the existence of both the _Document Retrieval_ and the _Response Generation_ in-house models for retrieving documents and generating responses. Our primary focus is on developing an effective Query Planner component. The models employed in the processes described in Sections 3.1, 3.2, 3.3, and 3.4 all utilize the 60B parameter HyperCLOVAX Kim et al. (2021); Shin et al. (2022) as their backbone large language model.

In our setup, the _Document Retrieval_ model, functioning as Naver's in-house search engine, retrieves three related documents based on the query generated through the Tree-of-Questions (ToQ) and the Query Generator as discussed in Sections 3.1 and 3.3, respectively, from a question. Subsequently, the _Response Generation_ model processes these documents to generate the final response, denoted as \(R\).

## Appendix F Related Work

Initial advancements in long-form complex question answering (QA) based on large language models have leveraged the Chain-of-Thought (CoT) approach Wei et al. (2022). Attempts to enhance the performance of QA models through sophisticated prompting techniques have set the stage for further developments in this area Sun et al. (2022); Lazaridou et al. (2022); Yu et al. (2022); Khalifa et al. (2023). Building on this foundation, recent efforts have increasingly focused on utilizing retrieval-based approaches. These efforts aim to augment the factual knowledge inherent in LLMs with retrieval search results Nakano et al. (2023); Qin et al. (2023). Despite the significant progress made, these methods often face challenges in scenarios requiring multiple active retrievals.

In response to these challenges, research has shifted towards developing multi-time retrieval methods. A notable method in this category is retrieving additional information using previous context at predetermined intervals Khandelwal et al. (2019); Borgeaud et al. (2022); Ram et al. (2023). However, these methods can be inefficient due to their reliance on previously generated tokens for queries and the fixed nature of the retrieval intervals.

Another significant approach in the field of multi-time retrieval for QA involves decomposing comprehensive questions into smaller, more manageable sub-questions, which aids in targeted information retrieval Yao et al. (2022); Khot et al. (2022); Khattab et al. (2022); Press et al. (2022); Jiang et al. (2023). This strategy has shown increased efficiency in determining the timing of retrievals, leveraging the inherent knowledge of LLMs.

However, as highlighted by Huang et al. (2023), relying solely on the inherent reasoning capabilities of LLMs without external feedback can lead to performance degradation. Our study addresses this issue by focusing on the generation of queries within a RAG-based multi-hop reasoning QA system. Therefore, we propose an interactive and explicit evaluation method that assesses whether the queries generated are sufficient to answer user questions, thus ensuring the creation of more effective and reliable responses