# Leveraging Retrieval-Augmented Generation for Automated Smart Home Orchestration
## 4 Test Scenarios and Validation Results
### Quantitative Performance Evaluation  
To complement the qualitative observations above, we introduce quantitative, numerical metrics characterising the orchestrator's behaviour during practical validation runs. Performance data were collected using 32 GB RAM, running Windows 10 and Apache Felix 7.0.5 as the OSGi container.  
Figure 21: Component XML descriptor for the generated service bundle, specifying service references for both the light and speaker components orchestrated in the user request.  
Figure 22: Apache Felix runtime logs showing successful activation and execution of the generated service bundle’s coordinated device actions.  
Figure 23: Service registration output in the OSGi Service Registry confirming the successful registration of the LivingRoomAutom bundle and its service dependencies.  
Measurement Methodology
* Context build time: from receiving user input to completion of prompt construction.
* Code generation time: duration of Java code generation via the LLM.
* Java compilation and packaging time: total time to compile Java code and package it into an OSGi JAR.
* Memory Consumption: peak resident-set-size (RSS) after context generation and after bundle activation, measured with _psutil_.
* Reliability: proportion of successful orchestrations across repeated runs.  
All time measurements used Python's high-resolution timer (_time.perf_counter()_), and memory was sampled via _psutil.Process().memory_info().rss_. Reliability statistics were computed over 20 automated runs using our batch_test.py harness: each run was marked successful only if the LLM generated bundle compiled, deployed in Apache Felix, and executed all intended device actions without exception. Table 2 shows the summary of results obtained.  
As described in Table 3, reliability was measured by executing the complete orchestration pipeline in a fully automated loop. From 20 trials:  
As shown in Figure 25, code generation accounts for 56.0% of the total pipeline time, while context build, compilation, and packaging contribute 22.2%, 15.3%, and 6.5%, respectively. This distribution underscores that LLM processing is the primary latency consumer, whereas compilation and JAR packaging introduce relatively minor overheads. Memory consumption remains stable around 416 MB between the end of prompt assembly and post-activation, demonstrating a consistent resource footprint suitable for edge gateway deployment. Over 20 test runs, the orchestrator achieved a 90% success rate (18/20) with a standard error of \(\pm\)6.7% (\(\pm\)13.2% at 95% confidence), indicating robust execution across repeated automated scenarios.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Evaluation Parameter** & **Mean \(\pm\) SE** & **Description** \\ \hline Context build time & 1.51 s \(\pm\) 0.32 s & Prompt complexity affects variability. \\ Code generation time & 3.81 s \(\pm\) 0.44 s & Dominant LLM latency contributor. \\ Java compilation time & 1.04 s \(\pm\) 0.12 s & Compiles vary with interface count. \\ JAR packaging time & 0.44 s \(\pm\) 0.07 s & Generally stable across all cases. \\ Memory (post-context) & 415.91 MB \(\pm\) 4.85 MB & Higher when more devices are included. \\ Memory (post-activation) & 416.40 MB \(\pm\) 5.02 MB & Slightly increases with multi-device bundles. \\ \hline \hline \end{tabular}
\end{table}
Table 2: Orchestrator performance metrics: execution time and memory usage.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Reliability Metric** & **Value** \\ \hline Success count & 18/20 \\ Reliability pp & 90\% \\ Standard error (SE) & \(\pm\)6.71\% \\
95\% CI margin of error & \(\pm\)13.15\% \\ \hline \hline \end{tabular}
\end{table}
Table 3: Reliability metrics based on 20 executions: success rate and error margins.