# LoRA-Tuned Multimodal RAG System for Technical Manual QA: A Case Study on Hyundai Staria
## 5 Experimental Results and Analysis
### Evaluation Methodology  
To rigorously assess the practicality, usability, and domain suitability of the proposed multimodal RAG system, a structured three-phase evaluation was conducted with 20 active-duty military maintenance personnel. The participant pool comprised 12 enlisted soldiers (ranks E-3 to E-6) and 8 officers (ranks O-1 to O-4), each with over one year of hands-on experience using conventional PDF-based technical manuals in operational maintenance settings. The evaluation was structured into three distinct phases. In Phase 1, titled \"System Orientation\" (30 min), participants received standardized demonstrations coveringsystem functionality, including Simple QA, RAG QA, and Multi-Turn QA. They were then given hands-on experience with the system using real-world queries derived from the Hyundai Staria maintenance manual. Phase 2, \"Structured Survey Assessment\" (45 min), required each participant to solve 10 representative maintenance scenarios using the system and rate its performance across four core dimensions using a five-point Likert scale (1 = strongly disagree, 5 = strongly agree). The evaluated dimensions included response speed (i.e., the timeliness of answer generation), information clarity (i.e., the ease of understanding and completeness of system output), system reliability (i.e., confidence in accuracy), and usability (i.e., interface intuitiveness and suitability for field deployment). Phase 3, \"Semi-Structured Interviews\" (15 min), involved gathering open-ended feedback from participants regarding system strengths, shortcomings, and suggestions for future enhancements. The overall satisfaction score averaged 4.4 out of 5 (SD = 0.6). Dimension-specific scores were 4.6 for response speed, 4.5 for information clarity, 4.3 for usability, and 4.2 for system reliability. Enlisted personnel reported slightly higher satisfaction (4.6) than officers (4.2), possibly due to the system's simplicity and intuitiveness. These results indicate that the proposed system offers significant advantages over traditional PDF-based manuals in terms of information retrieval speed and user comprehension. To validate annotation consistency, interrater reliability was assessed using 50 randomly selected image-text pairs independently reviewed by two expert annotators. Semantic alignment agreement was 96.3%, with a [PERSON]'s \(\kappa\) of 0.89, indicating excellent reliability. Qualitative feedback obtained from interview transcripts revealed recurring themes. Forty percent of participants suggested functional enhancements such as audio output support for low-light conditions and improved glossary definitions for technical terms. Thirty percent expressed concerns about fully relying on AI-generated answers without access to traceable manual references. Twenty percent proposed improvements in image-text linkage and visual annotation accuracy. The remaining ten percent highlighted the system's potential as a training tool for onboarding new maintenance personnel. While the evaluation demonstrated strong user satisfaction and operational promise, it was limited to a relatively small sample size and a single domain (automotive maintenance). Future work should include broader-scale evaluations across multiple equipment types and operational conditions to assess generalizability and robustness. To rigorously assess the practicality, usability, and domain suitability of the proposed multimodal RAG system, a structured three-phase evaluation was conducted with 20 active-duty military maintenance personnel. The participant pool comprised 12 enlisted soldiers (ranks E-3 to E-6) and 8 officers (ranks O-1 to O-4), each with over one year of hands-on experience using conventional PDF-based technical manuals in operational maintenance settings.