## 6 Evaluation
### Threats to Validity  
In the section we discuss the main internal (intrinsic to the experiment, e.g., systematic error) and external (related to the generalizability) threats to the validity of the evaluation results, and for each one we explain how we mitigated it. Regarding the correctness of the application-level behavior in the experiment: changes in the pollution level get reflected in the maximum allowed temperature setpoint for households, and the time after which the change becomes effective is subsequent to the moment sensors detect such change. Guarantees in this sense come from the structure of the application program itself. In fact, it functionally builds the output field (i.e., the field of maximum allowed temperature) through a field computation taking a single input field, namely the field of pollution level. The network performance that we measure as mean network delay, is dependent on all the free variables. For each such variable, we picked a set of values that ranges from smaller-than-normal to larger-than-normal, of course also experimenting with values that could be reasonably expected for a real deployment. To pinpoint the effect of every single variable on the performance, we execute simulation replicas for every combination in the Cartesian product of variable values, hence testing every combination. This allowed us to isolate the behavior and impact of every single change of any variable value, moreover enabling the analysis of interactions among multiple values (e.g., possible resonant phenomena). Simulations have been performed using state-of-the-art best practices, for instance, multiple randomization-controlled simulation repetitions (ten for each combination of the free variables), guaranteeing both a reasonable problem space exploration and complete reproducibility. The first version of [23] has been used in the past to simulate realistic LoRa communications between LoRa motes and LoRa gateways. Network communications other than those based on the LoRa protocol are modelled using a refined version of the network model proposed in [72], as described in Section 6.1. The most obvious threat to the validity of the network model is the lack of a model of lost packets, along with the assumption that the MQTT broker is behaving correctly and not having networking issues. In case of deployment in networks with low reliability, we expect the deployed system to show performance possibly different from the ones measured in simulation. However, we note that networking among households, edge, and cloud is usually reasonably reliable, and in our simulated  
Figure 12: Operating cost estimation for different deployment schemes. \(x\)-axes of both charts show the fraction of devices being hosted on the local thermostat, as opposed as on the cloud: every point is a deployment scheme that has been exercised retaining functionality, right-hand points are deployments with progressively more component moved to the edge of the network. Pulverization allows for exercising different deployment schemes without any change to the functional specification. Different lines on the eff-hand chart show projected pricing for hosting the broker on the self-hosted edge servers or moving it on the cloud. The right-hand side chart shows a breakdown of the overall pricing into its constituent components. The forecasted usage for the broker does not overcome the first pricing level; it is thus a constant contribution. Idle time power drain dominates the electricity consumption of smart thermostats (although the line looks flat, it imperceptibly grows); this suggests that moving computation towards the edge is generally cheaper, and suggests that better power saving strategies than keeping the device idle should be devised. Cloud hosting pricing decreases when devices are moved to the edge; since we request a new cloud instance every 100 thermostats hosted on cloud and pricing goes per-instance, the chart is discontinuous.