## 5 Implementation and Results
For example, chain-of-thought distillation, an advanced variant of this approach, involves generating step-by-step reasoning Question-and-Answer pairs. This method helps train smaller models to mimic not just the conclusions of the teacher model but also its reasoning pathways, improving the interpretability and reliability of the distilled models. \begin{table}
\begin{tabular}{l l l l} \hline \hline
**Score** & **Category** & **Part Number** & **Description** \\ \hline
0.923 & Fuse & 25860505 & Fuse Rejection Type 250 V 15 A Class RK5 \\
0.911 & Fuse & FRN-R-15 & Fuse Rejection Type 250 V 15 A Class RK5 \\
0.813 & Fuse & JTD015 & Fuse, Class J, Time Delay, 600 VAC, 15 A, 200 KAIC \\
0.745 & Fuse & KLDR015 & Fuse, Class CC, Time Delay, 600 VAC, 15 A, 200 \\  & & KAIC & \\
0.757 & Fuse & 349937133 & Fuse Rejection Time Delay 15 A Class CC \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Results and recommendations of the initial vector search. In this work, while a traditional distillation process was not employed, since the application used a pre-trained high-end LLM (GPT-4o), the distillation principles were leveraged in creating a curated dataset. This dataset, generated using a similar chain-of-thought methodology, was used to evaluate the effectiveness and accuracy of the application rather than to train a new model. Figure 10 illustrates an example of an advanced distillation process, highlighting the generation of chain-of-thought Question-and-Answer pairs. This evaluation approach ensures that the system's outputs align closely with the expected results and demonstrates the utility of distillation techniques for assessing LLM-based applications. The system was thoroughly evaluated using real-world manufacturing inventory data provided by the research sponsors. The dataset included a diverse range of part descriptions, both well-structured and poorly formatted, including entries with special characters or inconsistent terminology. This diversity was intentional to assess the robustness of the RAG-based system and the underlying language model when exposed to noisy, industry-specific input. To benchmark performance, a total of 200 Question-and-Answer pairs were generated using the language model. A representative subset is shown in Figure 11. The system responses were then evaluated using the DeepEval framework [37], a modern, open-source toolkit for assessing LLM outputs based on key quality dimensions. Figure 11: Excerpt of the Question and Answer pairs generated. Figure 10: Example of Few-Shot chain-of-thought Question/Answer pair. The evaluation focused on four core metrics: **Answer Relevance, Faithfulness, Context Recall**, and **Context Precision**. The results, summarized in Figure 12, revealed strong performance across all dimensions: 88.4% for Answer Relevance, 92.1% for Faithfulness, 80.2% for Context Recall, and 83.1% for Context Precision. These findings indicate that the system not only delivers highly relevant and factually accurate responses but also retrieves meaningful context while minimizing irrelevant noise. Although a subset of generated answers diverged from the expected phrasing, they generally conveyed the correct information and aligned well with user intent. These results affirm the effectiveness of our RAG-based pipeline for part lookup and recommendation tasks, with promising adaptability to other domains involving structured data and context-dependent retrieval. Table 3 summarizes and contrasts the core features of the traditional SQL search, generic machine learning-based methods, the recent RALLRec [38] framework, and our proposed LLM + RAG pipeline. The comparison highlights the capabilities of each approach in handling semantic queries, supporting context-aware recommendations, integrating privacy mechanisms, and operating in real-world industrial environments. Our method uniquely combines semantic understanding, advanced retrieval, and privacy features that are not present in other approaches. Figure 12: Results of evaluated metrics.