# Leveraging Retrieval-Augmented Generation for Automated Smart Home Orchestration
## 4 Test Scenarios and Validation Results
### Model Specialisation and Its Impact on Generated Code Quality  
While OpenAI's LLMs, such as GPT-4, offer strong general-purpose code generation capabilities, our experience shows that domain specialisation is essential for reliable orchestration in smart home systems. Without any structured prompt templates or contextual guidance, the model's raw outputs failed to compile or function in nearly all cases--demonstrating a near 100% failure rate. This confirms that generic LLMs, when used without domain awareness, are insufficient for generating compliant OSGi service bundles. In our setup, we define two categories of usage:  
* **Base LLM**: GPT-4 used without any domain-specific prompting, examples, or contextual guidance. This reflects the \"out-of-the-box\" usage scenario.
* **Specialised LLM**: GPT-4 guided using structured prompt templates, enriched with project-specific examples (e.g., interface names, method descriptions, OSGi DS structure), simulating the effect of a fine-tuned model. While we did not perform full fine-tuning, we evaluated the effect of simulated specialisation by feeding the model curated examples from our own orchestration codebase. This included: 1. Manufacturer-provided Java interface definitions and compiled bundles; 2. Existing OSGi Declarative Services (DS) annotations, XML descriptors, and manifest files; 3. Historical prompts and successful AI-generated service bundles. To quantify the impact of this simulated specialisation, we ran a controlled experiment. GPT-4 was used to generate orchestration bundles for 20 distinct user requirements. The outputs were evaluated based on:
* **Compile-error rate:*
* Whether the generated Java code compiled on the first attempt.
* **Unsatisfied-reference rate:*
* The number of unresolved service bindings in Apache Felix (from SCR logs).
* **Manual edit time:*
* Time required by a developer to correct the code and successfully deploy the bundle. These results, summarised in Table 4, highlight dramatic improvements. With structured domain prompts, the model reliably generated code that compiled, deployed, and operated with minimal or no manual correction. Specifically, the compile-error rate dropped from nearly 100% to 5%, and service reference errors (e.g., missing @Reference bindings or mismatched interface names) were almost entirely eliminated. Additionally, average  
Figure 25: Proportional time distribution chart across orchestration phases including context building, code generation, compilation, and JAR packaging.  
human correction time was reduced by over 80%, with many generated bundles requiring no intervention at all.  
The improvement stems from several benefits introduced through specialisation. The model consistently used correct package names, OSGi annotations, and device interface references, which previously were common points of failure. It also maintained naming conventions and structure aligned with our DS component model, improving code clarity and compatibility. Importantly, the specialised prompts helped the model manage complex multi-device orchestration scenarios more effectively--reducing logical errors like out-of-order method calls or incorrect sequencing of device actions.  
Therefore, even without retraining the model, simulating domain specialisation via tailored prompt templates and curated examples substantially improves both the quality and robustness of generated bundles. These results suggest that future work involving actual fine-tuning could further enhance performance, but that even prompt-level specialisation is highly effective and essential for deployment-grade orchestration.