Venue: NAACL 2025

# AutoKB: Automated Creation of Structured Knowledge Bases for

Domain-Specific Support

Rishav Sahay, Arihant Jain1, Purav Aggarwal, Anoop Saladi Amazon

Footnote 1: Equal contribution

###### Abstract

Effective customer support requires domain-specific solutions tailored to users' issues. However, LLMs like ChatGPT, while excelling in open-domain tasks, often face challenges such as hallucinations, lack of domain compliance, and generic solutions when applied to specialized contexts. RAG-based systems, designed to combine domain context from unstructured knowledge bases (KBs) with LLMs, often struggle with noisy retrievals, further limiting their effectiveness in addressing user issues. Consequently, a sanitized KB is essential to ensure solution accuracy, precision, and domain compliance. To address this, we propose AutoKB, an automated pipeline for building a domain-specific KB with a hierarchical tree structure that maps user issues to precise and domain-compliant solutions. This structure facilitates granular issue resolution by improving real-time retrieval of user-specific solutions. Experiments in troubleshooting and medical domains demonstrate that our approach significantly enhances solution correctness, preiseness, and domain compliance, outperforming LLMs and unstructured KB baselines. Moreover, AutoKB is 75 times more cost-effective than manual methods.

+
Footnote †: leftmargin=*]Equal contribution

+
Footnote †: leftmargin=*]Equal contribution

+
Footnote †: leftmargin=*]Equal contributions

## 1 Introduction

**Customer Support Agents (CSAs)** are chatbots Nuruzzaman and Hussain (2018); Xu et al. (2017) designed to resolve domain-specific user issues by providing customized, rule-compliant solutions1 aligned with domain standards. The advent of LLMs like ChatGPT OpenAI (2024); Ouyang et al. (2022) has revolutionized conversational AI, enabling it to handle diverse, open-domain queries with exceptional fluency. However, CSAs, such as product troubleshooting bots or medical assistants, face distinct challenges that demand precise, context-aware solutions for provided issues. While LLMs offer remarkable general-purpose capabilities, relying solely on them risks producing incorrect or generic solutions, limiting their effectiveness in these specialized roles.

Footnote 1: In CSAs, we refer user queries as **issues** and responses as **solutions**

To address these shortcomings, RAG Lewis et al. (2021) has emerged as a promising framework for building CSAs, by grounding LLM responses in retrieved domain-specific knowledge. However, the performance of RAG applications is largely dependent on the quality of the backend KB being used. Unstructured KBs, while covering a wide range of topics, are prone to noise and irrelevant information. In contrast, structured KBs resolve these issues by incorporating specific solutions and supporting domain constraints, enabling more precise and reliable knowledge grounding. RAG using unstructured KBs face additional challenges like token length limitations in LLMs, and difficulties in dynamically enforcing domain rules (because of unverified content in KB).

A significant limitation of existing approaches is their inability to provide solutions with the appropriate granularity. For instance, in troubleshooting scenarios, addressing a generic issue like Bluetooth

Figure 1: Illustration of an issue subtree for the seed issue _My headphones do not connect to phone_ with two child issues shown along with their respective metadata

connectivity problem is insufficient when the user faces a specific problem, such as Bluetooth not pairing with a device running an older Android version. Generic solutions that fail to address the user's exact issue often result in dissatisfaction.

Additionally, ensuring compliance with domain-specific policies--such as safeguarding rules for medical guidance or hardware safety instructions in product support--remains challenging in real-time solution generation. However, techniques like Chain-of-Thought Wei et al. (2023) and Reflexion Shinn et al. (2023) can enhance rule adherence but they may exacerbate input length constraints and increase latency, thereby complicating their practical application.

To address issues with solution accuracy, specificity, and domain compliance in existing systems, this paper introduces AutoKB, an automated pipeline for constructing a structured KB for **any domain**. As shown in Figure 1, the proposed KB employs a hierarchical tree structure where nodes represent issues at varying levels of granularity. Root nodes cover broad, generic issues, while child nodes capture more specific ones, each linked to solutions tailored to their level of detail. Node relationships are defined by **metadata** differences, ensuring coverage of both generic and specific user issues.

Following are the contributions of our work:

* We propose AutoKB, an **automated pipeline** that builds a KB, mapping issues to solutions, enriches them with domain knowledge, and ensures domain compliance through safeguarding rules.
* We introduce a **two-level tree-structured KB**, categorizing issues into generic and specific levels, differentiated using metadata. Each issue node is linked to solutions that match its required level of granularity.
* We develop a **hybrid retrieval strategy** that combines semantic and metadata-based search, significantly enhancing retrieval quality within CSAs utilizing the KB structure.

## 2 Related Work

Knowledge-based support systems aim to provide accurate, specific, and safe responses to user queries. The existing approaches can be broadly categorized into two main types: Prompting techniques for LLMs and RAG systems, which rely on underlying KBs.

Various prompting techniques have been developed to enhance LLM performance, particularly in rule-following, reducing hallucinations, and providing specific solutions. Chain-of-Thought (CoT) prompting Wei et al. (2023) and its variants like CoT with In-Context Learning (CoT-ICL) Dong et al. (2024) have shown promise in improving reasoning and rule-following capabilities. However, these methods still rely heavily on the LLM's pre-trained knowledge and may not provide grounded, specific, and safeguarded responses Zhao et al. (2024). RAG systems Lewis et al. (2021) combine the power of pre-trained language models with additional information, typically using retrieval methods to fetch relevant content and augment LLM responses. While RAG systems can improve response grounding and quality, their effectiveness is highly dependent on the quality and structure of the underlying KB.

KB construction approaches can be broadly categorized into two types: Unstructured and Structured. Unstructured KBs, built using web crawlers Huang et al. (2024) on popular search engines Caramancion (2024) and Databases Jing et al. (2024), cover a wide range of topics but often suffer from noise and irrelevant information. Structured KBs Hu et al. (2024); Kommineni et al. (2024) excel at representing domain-specific factual information and relationships between entities. However, both types face challenges in addressing specific user issues. Our KB framework develops a hierarchical tree-based structure capable of accommodating specific complex user issues and their solution knowledge, bridging the gap between issue representation and solution retrieval.

## 3 Proposed Methodology

### Knowledge Base Structure

We propose a hierarchical KB structured (Figure 1 as a two-level tree, where the root node represents generic issues, and child nodes represent more granular and specific issues. We term such a tree an **Issue Subtree**, which comprises a Parent Issue and its corresponding Child Issues (issues and nodes used interchangably). Each issue in the subtree is linked to a solution tailored to its granularity level. This structured approach enables the effective handling of highly specific customer issues while also addressing broader, more generic user concerns 

To differentiate between the granularity of parent and child issues, we utilize **Metadata** which is defined as a mapping of attributes to their respective values, capturing the key characteristics of an issue. For instance, for the troubleshooting example in Figure 1, the metadata for the child issue, _Wired headphones not connecting to Windows laptop_ is _[Connectivity: Wired, Secondary Device: Laptop, Secondary Device OS: Windows]_. These metadata keys (referred to as attributes), along with their possible value sets are predefined by domain experts for a given domain and referred to as the _attribute configuration_ as shown in Table 4 in Appendix.

For any issue, metadata is extracted based on the attributes defined in the attribute configuration using an LLM (see Prompt G.2). The extracted metadata helps differentiate between parent and child issues wherein parent issues exhibit broader attribute values, while child issues have more specific attribute values.

### Knowledge Base Creation

The automated KB creation process requires four key inputs for a given domain: 1) **Raw Issues**, which are historically observed user issues; 2) **Attribute Configuration**, defining attribute keys and their plausible values (detailed in Table 4 in Appendix); 3) **Domain Rules**, which outline the guidelines and constraints the KB must follow (examples in Table 5 in Appendix); and 4) **Domain Knowledge**, comprising unstructured documents such as user manuals and FAQs that can be utilized to build the KB.

KB creation pipeline consists of 3 modules, as illustrated in Figure 2 and outlined in Algorithm 1.

#### 3.2.1 Domain Knowledge Chunking (DKC)

To effectively utilize domain knowledge for issue resolution, we process unstructured documents and store it in a database. Initially, documents are divided into fixed-length chunks of 2048 characters, following Finardi et al. (2024). However, recognizing the limitations of traditional chunking methods, such as loss of coherency and context Dong et al. (2023), we introduce a novel technique called **Contextualized Chunking**.

Existing approaches, such as context-aware chunking and semantic chunking (Pinecone, 2025), focus on optimizing chunk boundaries but do not enrich individual chunks with additional contextual information critical for retrieval. Our approach, in contrast, generates a contextualized version for each chunk by incorporating information from preceding chunks using an LLM. The process, detailed in Prompt G.5, involves inputting the previous contextualized chunk as context and current chunk to the LLM to create an enriched, contextualized version. This method ensures that each chunk contains both local knowledge and a global understanding of the document, thereby enhancing retrieval accuracy. The original chunk and its contextualized version are then concatenated and encoded using a text encoder and stored in a VectorDB, as illustrated in Figure 2 and Algorithm 1.

#### 3.2.2 Issue Processing (IP)

To address the presence of duplicates in Raw Issues, we employ a two-step process of theme-based classification and clustering for de-duplication. First, we use Prompt G.6 to identify unique issue themes and Prompt G.7 to assign themes to each raw issue. Within each theme, we apply a clustering algorithm (detailed in Appendix D) to group similar issues, selecting cluster centroids as representative _Seed Issues_. This approach ensures a diverse and non-redundant set of issues for further processing.

Each Seed Issue is then transformed into an issue subtree using LLM Prompt G.8, which takes the seed issue and domain attribute configuration as input. This hierarchical structure, comprising a parent issue and its corresponding child issues, allows for a more nuanced representation of specific issues and their potential solutions. Figure 1 illustrates this process for a troubleshooting domain issue demonstrating how the initial seed issue is expanded into an issue subtree, while Figure 2 (bottom-left) and Algorithm 1 Lines 6-20 outline the complete workflow.

Figure 2: The KB creation pipeline comprising of 1) Domain Knowledge Chunking (DKC), 2) Issue Processing (IP), and 3) Solution Generation (SG).

 



#### 3.2.3 Solution Generation (SG)

The solution generation process employs RAG to link each node in the Issue Subtree to its corresponding solution. This approach treats issues in issue nodes as queries for retrieval from the previously constructed contextualized VectorDB and employs cosine similarity of retrieval. The top-5 retrieved chunks serve as input to an LLM prompt (G.9) to generate relevant solutions. To ensure the quality and appropriateness of the generated solutions, we focus on three key aspects: (1) **Correctness**, achieved through RAG and contextualized chunking, which improves retrieval recall and grounds the solutions in retrieved information;(2) **Domain Rule Compliance**, ensured by incorporating domain-specific rules into the prompt, guiding the model to adhere to defined constraints; and (3) **Granularity Alignment**, maintained by providing the issue and its metadata as input to the prompt, explicitly guiding the model to generate solutions that correspond to the issue's level of granularity. The steps is detailed in Algorithm 1 Lines 21-28.

### Hybrid Retrieval

We propose a retrieval strategy to integrate a structured KB with a CSA for resolving real-time user issues. The KB, organized with issue nodes linked to solutions, enables issue-issue matching. Solutions associated with the matched issue are retrieved and presented to the user. This strategy combines _soft_ semantic search for relevance with _hard_ metadata-based search for precision. Semantic search computes cosine similarity between text embeddings to identify the most semantically relevant nodes, while metadata-based search matches the query's metadata with the node metadata, ensuring precise retrieval. The goal is to find a node where the metadata closely matches the query--ideally an _exact-match_. If no exact match is found, nodes with metadata forming a superset of the query's (parent node) are considered _subset-matches_. However, if there is any conflict between the query and node metadata, the node is considered a _mis-match_ and excluded from the results. This strategy ensures 1) precise retrieval of solutions that match the query's granularity (in the case of exact match), and 2) broader solutions when a subset-match occurs.

To perform retrieval, as described in Algorithm 2 in the Appendix, each issue node (parent and child) in the KB is indexed using a text encoder and stored in a VectorDB, along with its metadata and solutions. When the CSA receives a real-time customer query, it extracts the metadata using an LLM with the prompt G.2. We make the assumption here that the CSA fully understands the issue by querying the customer effectively to establish the relevant metadata attributes before initiating KB retrieval. Once this is achieved, the query is encoded, and a semantic search is conducted over the indexed issues in the VectorDB. Based on the top \(k\) retrieved issues from the semantic search (referred to as the _target issue_), the following scenarios are handled:

1. **exact-match:** If a target issue's metadata is same as the query's metadata, the target issue is accepted.
2. **subset-match:** If the target issue is a Parent Issue and a _subset-match_, all its child nodes are traversed for an _exact-match_. If an _exact-match_ is found, the the child issue is accepted else the parent issue is accepted.
3. **no-match:** If the target issue is a Child Issue and a _no-match_, its parent and siblings are traversed. If an acceptable match is found (_exact-match_ or _subset-match_), the respective issue is accepted. Else, the target issue is discarded.

This hybrid retrieval strategy, illustrated in Figure 3, improves recall by addressing inaccuracies in semantic search. Metadata matching ensures exact alignment with the query, while the search among siblings and children enhances recall by covering overlooked matches by the semantic search.

## 4 Experimental Setup

### Datasets

We evaluate our approach on two distinct domains: **Troubleshooting** and **Medical Assistance**. For

Figure 3: Hybrid search strategy that utilises the proposed tree structure to perform metadata search on top of the node identified by the semantic search. The green node represents the subset-match, grey node represents the subset-match and the node with red glow represents the node matched using semantic search.



the troubleshooting domain, the KB is built using historical customer-reported issues from an e-commerce store, supplemented with domain knowledge extracted from user guides and product manuals. Due to proprietary constraints, we utilize a sampled subset of data from a real-world e-commerce store to mitigate any risks associated with sensitive information.

For the Medical Assistance domain, we treat patient symptoms as customer issues and corresponding treatments as solutions. The user symptoms are sourced from Kaggle (2016) and the domain knowledge is sourced from the dataset introduced in Shah et al. (2021). For both the domains, we generate the attribute configuration and domain rules using _claude-3-haiku_(Anthropic, 2023) as shown in Table 4 in Appendix.

We utilized these data sources for the KB creation process described in Section 3.2. Table 1 summarizes the details of the datasets and relevant statistics from the KB, including the number of raw, parent, child issues and total solutions.

### KB Creation Baselines

To evaluate the effectiveness of our KB creation process, we established baselines and ablated different modules: 1) **LLM-WK**, where the KB is created using the world knowledge of LLMs with the prompt in G.1 and user issues as input; 2) **Raw**, utilizing raw unstructured content from domain knowledge in chunks; 3) **Raw+CC**, leveraging contextualized chunks derived from domain knowledge; and 4) **AutoKB**, constructed using our proposed Issue Processing (IP) approach (Section 3.2.2), including both vanilla semantic search on child issues and a hybrid retrieval (**HR**) strategy on parent and child issues. We leverage _claude-3-haiku_ LLM for all of our KB creation tasks and _cohere.embed-multilingual-v3_(Cohere, 2023) as text-encoder.

### Evaluation Setup and Metrics

Our evaluation setup assesses the quality of the KB independently. Additionally, we evaluate the retrieval performance when the KB is integrated with a CSA for serving real time user issues. Due to confidentiality in the troubleshooting domain, we present relative improvements rather than absolute numbers.

#### 4.3.1 KB Quality Assessment

To assess the quality of our KB, we employed three metrics corresponding to the aspects presented in solution generation (see Section 3.2.3).

1. **Correctness** (\(Q_{C}\)): Measures the percentage of KB solutions that are correct with respect to the issues using human annotations (details in Appendix B).
2. **Domain Compliance** (\(Q_{D}\)): Evaluates the percentage of KB solutions that adhere to domain rules. This is done using _claude-3-sonnet_(Anthropic, 2023) with Prompt G.3.
3. **Metadata Granularity** (\(Q_{M}\)): Quantifies the granularity of solution based on its metadata in the KB. It uses an _Attribute Granularity Score_ (AGS) computed as the reciprocal of the number of possible values for a particular attribute of an issue. As an example the set of values for the attribute "Connectivity" of the Parent node in Figure 1 is Laptop and Phone and the AGS is thus equal to 0.5. The overall \(Q_{M}\) is the average AGS across all attributes: \[Q_{M}=\frac{1}{n}\sum_{i=1}^{n}\left(\frac{1}{\#\text{values}_{i}}\right)\] where \(n\) is total number of attributes as defined in the _attribute configuration_ and \(\#\text{values}_{i}\) is number of possible values for the \(i\)-th attribute for the solution.

#### 4.3.2 Retrieval Performance

To evaluate the retrieval effectiveness of our KB, we tested the retrieved content against a set of queries. We curate different variations of inputs from child issues using an LLM. These variations simulate different levels of ambiguity when interacting with the KB. Examples of these variations are shown in Table 6 in Appendix. To generate these variations, we employed a _claude-3-haiku_ LLM using the prompt illustrated in G.4.

To evaluate the KB's ability to retrieve relevant content, we employ the HitRate@\(k\) metric. This metric measures proportion of queries for which the relevant content is retrieved within the top \(k\) results.

\begin{table}
\begin{tabular}{l|c c c c} \hline \hline
**Domain** & **\#RJ** & **\#PI** & **\#CI** & **\#Sol** \\ \hline Troubleshooting & 265 & 49 & 482 & 2595 \\ Medical & 302 & 70 & 866 & 5196 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Knowledge Base Statistics. RI: Raw Issue, PI: Parent Issues, CI: Child Issues, Sol: Solutions 

We calculate HitRate@\(k\) for \(k\in\{1,3,5,10\}\), using the child issue or chunk from which the query is derived, as the ground truth.

## 5 Results and Analysis

**Retrieval Performance Analysis:** Table 2 summarizes the retrieval performance across various \(k\) values for different KB variations. The results indicate that contextualized chunking (CC) enhances HitRate by providing improved context for identifying the issue during retrieval. Structuring the KB using our approach (IP) significantly boosts retrieval performance by enabling direct embedding comparisons within the issue space rather than the issue-chunk space. Additionally, employing hybrid retrieval (HR) over the issue subtree, which combines semantic and metadata-based search, further improves retrieval outcomes.

**KB Quality Results:** Table 3 presents a comparison of different KBs across various quality metrics. AutoKB approach consistently outperforms both the LLM-WK and Raw KB baselines. In terms of correctness (\(Q_{C}\)), our KB achieves the highest scores, attributed to its groundedness enabled by RAG. Raw KB performs moderately well, particularly in the troubleshooting, while LLM-WK solutions lead to the most incorrect results due to their reliance on world knowledge, which can result in hallucinations. For domain compliance (\(Q_{D}\)), our approach achieves near-perfect scores, outperforming both Raw KB and LLM-WK. This indicates that our KB provides responses that are domain compliant. Furthermore, the high Metadata Granularity metric (\(Q_{M}\)) of our KB compared to other baselines showcases the superior granularity of solutions in our KB. Figure 4 provides a qualitative comparison, highlighting how LLM-WK generates a generic and domain-noncompliant solution (marked in red), whereas AutoKB offers a more specific and domain-compliant solution (marked in green) for the issue of AirPods.

## 6 Industry Impact

AutoKB demonstrated practical effectiveness and scalability in a large e-commerce context. (1) In self-serve troubleshooting, the KB offered curated solutions for 7K issue-solution pairs across 6 products, achieving a 95% acceptance rate from human annotators. (2) During a 4-week A/B test with a Troubleshooting CSA across 6 product categories, AutoKB helped reduce the return rate and improved chatbot adoption compared to an internal baseline using manually curated KB.

Cost comparisons revealed significant savings. Creating a KB for 265 troubleshooting issues with _claude-3-haiku_ cost $6.69 (details in Appendix C), while human experts, at $3.75/hour and 0.5 hours per issue, would cost $496.87. This demonstrates that our approach is 75 times more cost-effective, showcasing its potential to lower costs in knowledge-based support systems.

## 7 Conclusion

In this paper, we propose AutoKB, an automated strategy for curating structured KBs to deliver cor

\begin{table}
\begin{tabular}{r c|c c|c c c c} \hline \multicolumn{2}{l|}{**Domain**} & **KB** & **CC** & **IP** & **HR** & **HitRate@1** & **HitRate@3** & **HitRate@5** & **HitRate@10** \\ \hline \multirow{3}{*}{Troubleshooting} & Raw & & & & - & - & - \\  & & ✓ & & +0.6 & +5.0 & +1.4 & +0.9 \\  & AutoKB & ✓ & ✓ & & +7.5 & +14.1 & +13.2 & +15.0 \\  & & ✓ & ✓ & ✓ & **+12.5** & **+17.3** & **+16.3** & **+18.0** \\ \hline \multirow{3}{*}{Medical} & Raw & & & & 52.0 & 57.2 & 63.9 & 75.7 \\  & & ✓ & & +6.7 & +8.1 & +8.1 & +6.1 \\ \cline{1-1}  & AutoKB & ✓ & ✓ & & +18.0 & +18.3 & +19.7 & +14.5 \\ \cline{1-1} \cline{2-7}  & ✓ & ✓ & ✓ & **+20.0** & **+22.3** & **+25.6** & **+18.8** \\ \hline \end{tabular}
\end{table}
Table 2: Comparison of retrieval performance for different KB configurations. CC: Contextualized Chunking, IP: Issue Processing, HR: Hybrid Retrieval. Results show incremental improvements relative to the first baseline in each domain.

\begin{table}
\begin{tabular}{r r|c c c} \hline \multicolumn{2}{c|}{**Domain**} & **KB** & \(Q_{C}\) & \(Q_{D}\) & \(Q_{M}\) \\ \hline \multirow{3}{*}{Troubleshooting} & LLM-WK & - & - & - \\  & Raw & +22.9 & +2.8 & +0.22 \\  & AutoKB & **+24.8** & **+5.0** & **+0.57** \\ \hline \multirow{3}{*}{Medical} & LLM-WK & 67.5 & 96.8 & 0.21 \\  & Raw & +24.6 & -0.1 & +0.13 \\ \cline{1-1}  & AutoKB & **+25.7** & **+1.3** & **+0.50** \\ \hline \end{tabular}
\end{table}
Table 3: Comparison of KB Quality Metrics. Incremental improvements are shown relative to the first baseline in each domain.

 

rect, domain-compliant, and issue-specific solutions. Our approach introduces a hierarchical KB, organized into parent and child issues, effectively addressing varying levels of granularity in user concerns using metadata. By leveraging contextualized chunking and RAG-based solution generation, we enhance the correctness of KB solutions. Experimental evaluations in troubleshooting and medical domains demonstrate that our approach outperforms traditional methods in solution quality and retrieval performance within a CSA.

## References

* (1)
* Amthorpic (2023) Anthropic. 2023. The claude 3 model family: Opus, sonnet, haiku. [https://www-cdn.anthropic.com/de8ba9b0l5ab7cbabdf5c3h3bb8db7bbc618857627/Model_Card_Claude_3.pdf](https://www-cdn.anthropic.com/de8ba9b0l5ab7cbabdf5c3h3bb8db7bbc618857627/Model_Card_Claude_3.pdf).
* Caramancion (2024) Kevin Matthe Caramancion. 2024. Large language models vs. search engines: Evaluating user preferences across varied information retrieval scenarios. _Preprint_, arXiv:2401.05761.
* Coherer (2023) Coherence. 2023. cohere-embed-multi.
* Dong et al. (2024) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. 2024. A survey on in-context learning. _Preprint_, arXiv:2301.00234.
* Dong et al. (2023) Zican Dong, Tianyi Tang, Lunyi Li, and Wayne Xin Zhao. 2023. A survey on long text modeling with transformers. _arXiv preprint arXiv:2302.14502_.
* Finardi et al. (2024) Paulo Finardi, Leonardo Avila, Rodrigo Castelloni, Pedro Gengo, Celio Larcher, Marcos Piau, Pablo Costa, and Vinicius Carida. 2024. The chronicles of rag: The retriever, the chunk and the generator. _Preprint_, arXiv:2401.07883.
* Hu et al. (2024) Yujia Hu, Shrestha Ghosh, Tuan-Phong Nguyen, and Simon Razniewski. 2024. Qptkb: Building very large knowledge bases from language models. _Preprint_, arXiv:2411.04920.
* Huang et al. (2024) Wenhao Huang, Zhouhong Gu, Chenghao Peng, Jiaqing Liang, Zhixu Li, Yanghua Xiao, Liqian Wen, and Zulong Chen. 2024. AutoScraper: A progressive understanding web agent for web scarger generation. In _Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing_, pages 2371-2389, Miami, Florida, USA. Association for Computational Linguistics.
* Jing et al. (2024) Zhi Jing, Yongye Su, and Yikun Han. 2024. When large language models meet vector databases: A survey. _Preprint_, arXiv:2402.01763.
* Kaggle (2016) Kaggle. 2016. Symptom disease sorting.
* Kommineni et al. (2024) Vamsi Krishna Kommineni, Birgitta Konig-Ries, and Sheeba Samuel. 2024. From human experts to machines: An llm supported approach to ontology and knowledge graph construction. _Preprint_, arXiv:2403.08345.
* Lewis et al. (2021) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktaschel, Sebastian Riedel, and Douwe Kiela. 2021. Retrieval-augmented generation for knowledge-intensive nlp tasks. _Preprint_, arXiv:2005.11401.
* Nuruzzaman and Hussain (2018) Mohammad Nuruzzaman and Omar Khadeer Hussain. 2018. A survey on chatbot implementation in customer service industry through deep neural networks. In _2018 IEEE 15th international conference on e-business engineering (ICEBE)_, pages 54-61. IEEE.
* OpenAI (2024) OpenAI. 2024. GPT-4 technical report. _Preprint_, arXiv:2303.08774.
* Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. _Preprint_, arXiv:2203.02155.
* Pinecone (2025) Pinecone. 2025. Chunking strategies. Accessed February 21, 2025.
* Shah et al. (2021) Darsh J Shah, Lili Yu, Tao Lei, and Regina Barzilay. 2021. Nutri-bullets: Summarizing health studies by composing segments. _Preprint_, arXiv:2103.11921.
* Shinn et al. (2023) Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. _Preprint_, arXiv:2303.11366.
* Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. _Preprint_, arXiv:2201.11903.
* Xu et al. (2017)Anbang Xu, Zhe Liu, Yufan Guo, Vibha Sinha, and Rama Akkiraju. 2017. A new chatbot for customer service on social media. In _Proceedings of the 2017 CHI conference on human factors in computing systems_, pages 3506-3510.
* Zhao et al. (2024)Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjiie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yin Nie, and Ji-Rong Wen. 2024. A survey of large language models. _Preprint_, arXiv:2303.18223 

Algorithm

Algorithm 1 outlines the step-by-step process of implementing the KB creation process. The algorithm consists of three main steps: Domain Knowledge Chunking, Issue Processing, and Solution Generation. It takes as input raw issues, attribute configurations, domain information, domain rules, and domain knowledge. The output is a structured Knowledge Base consisting of issue subtrees with associated solutions.

Algorithm 2 presents a hybrid retrieval strategy for integrating KB, combining semantic search with metadata-driven refinement. It takes a customer query, performs semantic search to retrieve top-k results, and then refines these results based on metadata matching. It navigates through parentchild relationships in the knowledge base, aiming to find the most relevant nodes that match the query's metadata.

## Appendix B Human Annotation Documentation

### Correctness Evaluation

We employed human experts as annotators for measuring the Correctness (\(Q_{C}\)) of our KB data. Annotators were provided with the instruction "_Your task is to check if the given solution is correct for the user issue. If the given solution is correct, respond with YES, otherwise NO."_ along with the user issue and the solution from the KB. We recorded the responses from human experts using a binary scoring system (1 for YES, 0 for NO) and report the average measure of correctness in Table 3.

### Validation of Automated LLM Evaluation

To validate our automated LLM evaluation approach for Domain Compliance Evaluation and Metadata Extraction Tasks, we conducted a comparative study between LLM-based evaluations and human assessments. We calculated the accuracy between the LLM-based evaluations and human assessments for each task. The results demonstrated high overall accuracy of 97% for Domain Compliance Evaluation task and 94% for Metadata Extraction task. The high accuracy numbers underscore the strong alignment between LLM-based evaluations and human judgment, supporting the reliability of LLM based evaluation.

### Annotation Details

Our annotation process varied by domain to ensure high-quality data. For the Troubleshooting domain, we recruited domain experts with experience in creating product troubleshooting content. For the medical domain, we utilized Amazon Mechanical Turk workers who met relevant qualification criteria. To measure inter-annotator agreement, we followed the standard protocol of performing dual annotations on a sample set of 10% of the data. We observed an agreement rate of 97% demonstrating the reliability of our annotations across all domains.

## Appendix C Cost Calculation

We calculate the cost and latency of generating a KB for the troubleshooting domain, comprising of \(265\) raw issues.

#### Breakdown of LLM Calls

The total number of LLM calls is as follows:

* **Issue Deduplication:**\(266\) calls (\(1\) for issue theme identification and \(265\) for issue theme assignment)
* **Issue Processing:**\(49\) calls (for issue subtree creation)
* **Contextualized Chunking:**\(1,000\) calls (approx \(100\) documents with \(10\) chunks each)
* **Solution Generation:**\(531\) calls (\(49\) for parent issues and \(482\) for child issues)
* **Solution-Metadata Detection:**\(832\) calls (one per generated solution)

Thus, the total number of LLM calls = 2678

#### Cost Calculation

The cost of KB generation is calculated using the following formula:

\[\text{Total cost}=N\times\left(\frac{T_{\text{in}}}{1000}\cdot C_{\text{in}}+ \frac{T_{\text{out}}}{1000}\cdot C_{\text{out}}\right)\]

where:

* \(N\): Total number of LLM calls
* \(T_{\text{in}}\): Average number of input tokens per LLM call (5K)
* \(T_{\text{out}}\): Average number of output tokens per LLM call (1K)
* \(C_{\text{in}}\): Cost per 1000 input tokens (0.00025$)
* \(C_{\text{out}}\): Cost per 1000 output tokens (0.00125$)

Substituting the values, the total cost of KB generation totals to 6.695$ 



## Appendix D Issue Clustering Details

We provide more details about the issue de-duplication algorithm along with the specific hyper-parameters being used.

1. We generate the issue embeddings using open-source (Cohere, 2023).
2. We use UMAP along with HDBCAN to cluster the issues within each themes. UMAP is used for dimensionality reduction.
3. UMAP : n_neighbors=15, min_dist=0.1, n_components=20, and metric='euclidean'
4. HDBSCAN : min_cluster_size=5 and metric='euclidean'

## Appendix E Additional Tables

Table 4 presents the pre-defined attribute configurations for the troubleshooting and medical domains. The attributes are designed to capture key information relevant to each domain, facilitating better issue diagnosis and solution recommendation.

Table 5 outlines the domain-specific rules applied in our system for troubleshooting and medical domains. These rules are crucial for ensuring that the system provides safe, ethical, and practical recommendations. They guide the system to avoid potentially harmful or unhelpful suggestions, maintaining user safety and enhancing the overall quality of assistance provided.

Table 6 demonstrates various phrasings of the same user input utilized in analyzing the performance on retriever



## Appendix F Qualitative Results

Figure 4 presents a comparative analysis of troubleshooting responses generated by AutoKB and LLM-WK. Our system demonstrates superior performance by providing more granular, step-by-step solutions (highlighted in green). In contrast, the baseline LLM-WK offers a less structured, all-at-once response. Additionally, our framework effectively identifies and filters out non-compliant information (highlighted in red) that violates domain-specific rules and should not be presented to customers.

Figure 4: Qualitative comparison of the response for the Issue: _Airpods won’t turn on when connected to a laptop_ for **LLM-WK** and the proposed **AutoKB**.Violations in domain compliance are highlighted in red and the granularity of the solutions is highlighted in green.





\begin{table}
\begin{tabular}{l l|c} \hline \hline
**Variations** & \\ \hline _I’ve an issue with my phone_ \\ _I’m facing a problem with my mobile._ \\ _There seems to be a problem with the device I use for communication, specifically my phone._ \\ \hline \hline \end{tabular}
\end{table}
Table 6: Examples of different variations of User Input _”I have an issue with my phone"_

\begin{table}
\begin{tabular}{p{108.4pt} p{52.7pt}|p{93.9pt}} \hline \hline
**Dataset** & **Rule \#** & **Description** \\ \hline Troubleshooting & 1 & _Avoid solutions that suggest use of abrasive cleaners or chemical solutions._ \\  & 2 & _Avoid solution that point the user to refer to the user manual._ \\  & 3 & _Avoid recommending solutions that asks the user go to the service center for repair or replace._ \\  & 4 & _Avoid recommending steps such as contacting product support and customer support._ \\ \hline Medical & 1 & _Do not recommend high-risk procedures._ \\  & 2 & _Avoid giving definitive medical diagnoses;_ \\  & 3 & _Refrain from recommending treatments that lack strong scientific evidence._ \\  & 4 & _Ensure recommendations consider user-reported allergies to avoid suggesting harmful treatments._ \\ \hline \hline \end{tabular}
\end{table}
Table 5: Domain specific rules for the different domains

```
1:Customer Query \(Q\), Vector Database (VectorDB), \(k\) retrieval value
2:AcceptedNodes (Set of Accepted Issue Nodes)
3:QueryMetadata \(\leftarrow\) ExtractMetadata(\(Q\))\(\triangleright\) Extract metadata using an LLM Prompt G.2
4:QueryEmbedding \(\leftarrow\) Encode(\(Q\))\(\triangleright\) Compute query embedding
5:TopKResults \(\leftarrow\) SemanticSearch(QueryEmbedding, VectorDB, \(k\))\(\triangleright\) Retrieve top-\(k\) issues from VectorDB
6:AcceptedNodes \(\leftarrow\{\}\)
7:for TargetIssue \(\in\) TopKResults do
8:TargetMetadata \(\leftarrow\) TargetIssue.metadata
9:if QueryMetadata = TargetMetadata then
10:AcceptedNodes \(\leftarrow\) AcceptedNodes \(\cup\) {TargetIssue} \(\triangleright\) Exact match found
11:else if QueryMetadata \(\subseteq\) TargetMetadata then
12:if TargetIssue.type = Parent then
13:found \(\leftarrow\) False
14:for ChildIssue \(\in\) GetChildren(TargetIssue) do
15:if QueryMetadata = GetMetadata(ChildIssue) then
16:AcceptedNodes \(\leftarrow\) AcceptedNodes \(\cup\) {ChildIssue} \(\triangleright\) Exact match in children
17:found \(\leftarrow\) True
18:break
19:endif
20:endif
21:endfor
22:if found = False then
23:AcceptedNodes \(\leftarrow\) AcceptedNodes \(\cup\) {TargetIssue} \(\triangleright\) Accept parent
24:endif
25:endif
26:else
27:if TargetIssue.type = Child then
28:Parent \(\leftarrow\) GetParent(TargetIssue)
29:if QueryMetadata \(\subseteq\) GetMetadata(Parent) then
30:found \(\leftarrow\) False
31:for Sibling \(\in\) GetChildren(Parent) do
32:if QueryMetadata = GetMetadata(Sibling) then
33:AcceptedNodes \(\leftarrow\) AcceptedNodes \(\cup\) {Sibling} \(\triangleright\) Accept sibling
34:found \(\leftarrow\) True
35:break
36:endif
37:endif
38:endif
39:endif
40:endif
41:endfor
42:return AcceptedNodes
```

**Algorithm 2** Hybrid Retrieval Strategy for KB Integration (Section 3.3) 

Prompts

**In****Function </ALL_tags>**: Prompt

**Instruction** You are a solution provider for a given user issue whose task it to provide solutions for a particular domain. You will be given as input the following pieces of information:

1. Domain Information: This is the information about the domain enclosed within the XML tags <domain_info>.
2. Issue: This is the issue the customer is facing . This is enclosed within the XML tags <issue>.

Instructions:

1. Enclose your response within the XML tags <response>.
2. Provide multiple possible solutions.
3. Enclose each solution within the XML tags <solution>.

**In-context examples:** Here are some examples:

<example> ... </example>

<example> ... </example>

**Input:** Now here is the input to you:

<domain_info> [domain_info] </domain_info>

<issue> [issue] </issue>

```
0: Template </ALL_tags>**: Propt

**Algorithm 5**  **The Complete Event 

* [10]**Instructions:**

As a text modifier, your role involves introducing subtle changes to a provided text snippet. This task requires adherence to specific types of alterations, categorized by difficulty levels. The types of permissible variations are as follows:

- Easy Variations: 1. Addition or removal of punctuation marks. 2. Utilization of different variants of the same lemma.

- Medium Variations: 1. Employment of synonyms for any word. 2. Phrase modifications: either by substituting a single word with a phrase or vice versa.

- Hard Variations: 1. Structural transformation of the text, entailing a complete reformulation while preserving the original message. Under no circumstances should changes deviate from these guidelines. The core message and structural integrity of the text must remain intact. The provided text will be enclosed within <original_text> tags. Your task is to generate five variations for each difficult level:

- For easy variations, enclose each variant within <easy_variations> tags, with individual variations wrapped in <variation> tags.

- For medium variations, use <medium_variations> for the group and covariation> for individual entries.

- For hard variations, group them under <hard_variations>, with each distinct variant in a <variation> tag.

**In-context examples:**

Here are some examples:

<example> ... </example>

<example> ... </example>

**Input:**

Now here is the input to you:

<contexv> {context} </context>

<original_texv> {original_text} </original_text> 

* [10] A. K. Krizhevsky, S. Ghahramani, and M. Welling. Learning from a large dataset to be useful for image recognition. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 235-246, 2017.
* [11] J. L. Friedman, R. Tibshirani, and C. Szegedy. Imagenet classification with deep convolutional neural networks. In _Advances in Neural Information Processing Systems (NIPS'98)_, pages 139-140, 1998.
* [12] D. F. Kingma and P. Ba. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6150_, 2014.
* [13] T. Haarblatt et al., "ImageNet: A survey on machine learning. ICLR, 2015.
* [14] Y. Bengio, V. LeCun, and Z. Bengio. Generative adversarial nets. In _Advances in neural information processing systems_, volume 10. Springer, 2016.
* [15] H. Bakhtinov, N. Khosla, and E. Rabinovich. Deep residual learning for image recognition. In _IEEE International Conference on Acoustics and Speech (ICASSP)_, pages 1-10, 2016.

[MISSING_PAGE_POST]

 

* [1]**Instructions: You are a Issue Tree Generator whose task it to create a issue tree based on issue attributes. You will be given as input the following:

1. Domain Information: This is the information about the domain within the XML tags <domain_info>.

2. Customer Issue: The issue related to the domain within cissue> XML tags.

3. Attributed Configuration: The various attributes and its possible values in general within <attr_config> XML tags. The root of the tree is a Parent Issue while the children of the parent issues are Child Issues.

In order to create the issues, you should do the following:

\(<\)general_instructions?

1. Analyse the attributes of the fed issue and within <thinking></thinking> try to find the generic version (in terms of attributes) of the issue known as the Parent Issue.

2. Create a Parent Issue which is more applicable to all kind of attribute values within <g>~/g>. Detect the attributes as well within <a>~/a>.

3. Now think within <thinking></thinking> again, what child issues which will be attribute specific are possible out of the generic issue. Each of the attribute keys should now take a single value.

4. You will create this specific attribute variations within <i></j>. Also predict the attributes of the issues within <a>~/a>.

5. Only create the valid candidates whose attribute combinations makes sense as per the attribute constraints.

6. Make sure that each of the attribute values reflect in the issues being created.

\(<\)general_instructions>

**In-context examples: Here are some examples:

\(<\)example\(>...\) &/example\(>\)

\(<\)example\(>...\) &/example\(>\)

**Input:**

Now here is the input to you:

\(<\)domain_info> {domain_info} !/domain_info>

\(<\)attr_config> {attribute_configuration} !/attr_config>

\(<\)tissue\(>\) {issue} !/issue>

```
0: From C92 Solution Generation Prompt
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
50:
51:
52:
53:
54:
55:
56:
57:
58:
59:
60:
61:
62:
63:
64:
65:
66:
66:
67:
68:
69:
70:
71:
72:
73:
74:
75:
76:
77:
78:
79:
80:
81:
82:
83:
84:
85:
86:
87:
88:
89:
90:
91:
92:
93:
94:
95:
96:
97:
98:
99:
100:
11:
12:
13:
14:
15:
16:
17:
18:
19:
19:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
51:
52:
53:
54:
55:
56:
57:
58:
59:
60:
61:
62:
63:
64:
65:
66:
67:
68:
69:
70:
71:
72:
73:
74:
75:
76:
77:
78:
79:
80:
81:
82:
83:
84:
85:
86:
87:
88:
89:
90:
91:
92:
93:
94:
95:
96:
97:
98:
99:
99:
100:
11:
12:
13:
14:
15:
16:
17:
18:
19:
19:
19:
10:
12:
14:
15:
16:
17:
18:
19:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
50:
51:
52:
53:
54:
55:
56:
57:
58:
59:
61:
62:
63:
64:
65:
66:
67:
68:
69:
70:
71:
72:
73:
74:
75:
76:
77:
78:
79:
80:
81:
82:
83:
84:
85:
86:
87:
88:
89:
90:
82:
85:
89:
86:
89:
80:
83:
87:
89:
80:
81:
82:
83:
85:
86:
87:
88:
89:
89:
91:
80:
81:
83:
85:
89:
81:
85:
86:
87:
89:
82:
89:
83:
85:
89:
86:
87:
88:
89:
89:
90:
80:
81:
82:
83:
85:
89:
86:
87:
88:
89:
89:
80:
81:
89:
82:
83:
85:
89:
86:
89:
87:
88:
89:
89:
80:
81:
89:
80:
82:
83:
85:
89:
81:
83:
85:
86:
87:
88:
89:
89:
80:
81:
82:
83:
85:
89:
86:
89:
87:
88:
89:
89:
80:
81:
89:
82:
83:
89:
83:
85:
89:
84:
85:
86:
87:
88:
89:
89:
89:
80:
81:
89:
82:
85:
89:
83:
89:
85:
89:
86:
89:
87:
89:
88:
89:
89:
80:
81:
89:
80:
82:
89:
83:
89:
80:
83:
89:
84:
85:
89:
86:
89:
87:
88:
89:
89:
89:
80:
81:
89:
80:
82:
89:
83:
89:
85:
89:
86:
89:
87:
89:
88:
89:
89:
80:
81:
89:
80:
82:
89:
83:
89:
89:
80:
83:
89:
81:
89:
82:
83:
89:
84:
85:
89:
86:
89:
87:
89:
88:
89:
89:
89:
80:
89:
80:
81:
89:
80:
82:
89:
83:
89:
89:
80:
83:
89:
81:
89:
82:
89:
83:
89:
84:
89:
85:
89:
86:
89:
87:
89:
89:
89:
89:

80:89:

80:89:

81:
89:
80:
82:
89:

83:
89:

81:
89:

82:
89:

83:

84:

85:

86:

87:

88:

8:

88:

8: