# Leveraging Retrieval-Augmented Generation for Automated Smart Home Orchestration
## 4 Test Scenarios and Validation Results
### Stress Testing and Scalability Evaluation  
To assess the orchestrator's performance under more realistic and demanding conditions, we conducted a stress test using 15 simulated smart home devices across different categories (smart lights, thermostats, motion sensors, speakers, and locks). This test aimed to evaluate how the system handles concurrent orchestration requests, manages device usage, and maintains performance under load. The test setup is detailed below, and the results are summarised in Table 5.  
* Devices: 15 simulated devices introduced through MQTT discovery.
* User Requirements: A total of 6 sequential orchestration requests followed by 4 simultaneous requests, each targeting different combinations of devices.
* Execution Method: Sequential requests were executed one after the other. The simultaneous requests were run in parallel using Python's _ThreadPoolExecutor_, simulating concurrent smart home events such as scheduled automations or overlapping voice commands Metrics collected:
* Orchestration Latency: Time from user input to bundle activation in Felix.
* Success Rate: Number of orchestrations completed without error.
* Memory Usage: Peak RAM usage during the orchestration cycle.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Metric** & **Base LLM** & **Specialised LLM** & **Improvements** \\ \hline Compile-error rate & \(\sim\)100\% & 5\% (1/20) & \(\sim\)95 percentage point \\ Unsatisfied-reference rate & \(>\)1.0 errors/run & 0.05 errors/run &
\begin{tabular}{c} Near elimination \\ \(\sim\)83\% reduction \\ \end{tabular} \\ Manual edit time (mean) & \(>\)30 s & 5 s & \(\sim\)83\% reduction \\ \hline \hline \end{tabular}
\end{table}
Table 4: Comparison between base and specialised LLM in code generation quality.  
* Device Contention Handling: Number of orchestration attempts delayed or retried due to conflicting device usage. During sequential execution, the orchestrator maintained a consistent response time of approximately 4.2 s per orchestration. When handling the 4 concurrent orchestration requests, latency slightly increased to an average of 5.9 s, primarily due to device selection conflicts and parallel bundle handling. Of the 10 total orchestration attempts, 9 completed successfully. One failure occurred due to a device contention issue (i.e., two services attempting to control the same device simultaneously), which was automatically detected and retried using the orchestrator's internal resource tracking logic (DeviceUsageTracker). This component ensures safe, thread-aware management of device availability. The system remained stable throughout the test, with memory usage peaking at around 448 MB, demonstrating suitability for deployment on lightweight home automation gateways. No unhandled exceptions, system crashes, or unstable service states were observed. These results show that the orchestrator scales well in moderately complex environments, handling multiple active devices and overlapping automation requests with minimal performance impact and high reliability. The test confirms the system's ability to coordinate device usage efficiently and recover gracefully from temporary resource conflicts.