## 3 Simulating MQTT-SN
### Environment Evaluation  
The implemented simulation framework allows us to perform several tests in order to evaluate the delivery performance obtained by MQTT-SN clients in different network conditions. In particular, the following two performance metrics can easily be computed: (i) the hitrate, defined as the percentage of the published MQTT-SN messages that successfully reach the subscribers, and (ii) the end-to-end latency from a message publication to its arrival at the subscribers. We run several tests where one single client transmits 1400-bytes publish messages at a fixed rate of 1 msg/s to 10 subscribers. Each test is characterized by different underlying network conditions, in terms of the Bit Error Rate (BER) of the wireless channel as well as the amount of traffic produced by interference nodes. Moreover, in each test we controlled the QoS requested by both publishers and subscribers to evaluate its effect on the delivery performance. Let \(Q_{p}\) and \(Q_{s}\) be the publisher and subscriber requested QoS levels, respectively. According to the MQTT standard specification, the QoS of notifications sent by a broker to a subscriber must be the minimum of the QoS of the originally published message and the maximum QoS granted by the broker to the subscriber. Therefore, only the couples where \(Q_{s}\leq Q_{p}\) have been tested. Figure 3 shows the hitrate value and latencies obtained for different QoS configurations and different underlying network conditions. Several considerations can be made by the inspection of such results:  
* For what concerns the hitrate, unsurprisingly, when the network conditions are favorable (no interference traffic and low BER), the delivery performance is good regardless of the QoS level used. When the BER or the level of interference traffic increase, all QoS configurations except for \(Q_{p}=2,Q_{s}=2\) suffer from a decrease in hitrate. Furthermore, it is evident how the hitrate is greatly dependent on the subscriber QoS level \(Q_{s}\), rather than \(Q_{p}\). Indeed, at all interference traffic levels, the curves sharing the same \(Q_{s}\) (blue, brown and green for \(Q_{s}=0\), orange and green for \(Q_{s}=1\)) follow the same trend at different BER.
* As for the end-to-end latency, similar observations can be derived. In the case of good network conditions (low BER and interference traffic), all QoS levels show in generally low latencies. Figure 3b,d,f again show a clustered behavior dominated by \(Q_{s}\), with configurations sharing the same subscriber QoS having the same trend. However, the gap in performance obtained by difference configurations with the same \(Q_{s}\) is evident: indeed, while at \(Q_{s}=1\) the end-to-end latency is bounded by the maximum number of retransmissions set by \(N_{retry}\), for \(Q_{s}=2\) the latency can increase indefinitely. Indeed, while at good network condition the impact of an higher QoS level is somehow limited (although always producing higher end-to-end latencies), at high BER and interference traffic levels using QoS 2 produce latencies, which are two order of magnitude greater than in QoS 0. This is clearly due to the very high number of retransmissions performed by the broker in the attempt to deliver messages to the subscribers.  
Figure 3: Average hitrate (**left**) and latency (**right**) for all publisher/subscriber QoS combinations under different network conditions (varying BER and Interference traffic).