# E-MQTT: End-to-End Synchronous and Asynchronous Communication Mechanisms in MQTT Protocol
## 5 Experimental Results  
We conducted a series of experiments to analyze the performance of E-MQTT. To facilitate a comprehensive evaluation, we drew a comparative analysis between the proposed E-MQTT QoS 3 and the existing MQTT QoS 2, given their closely aligned transmission processes. We implemented test clients (a publisher and subscribers) alongside broker applications that used both MQTT and E-MQTT because other benchmark frameworks or performance evaluation methods [56; 57; 58; 59] do not support the features of E-MQTT. The source codes of the test applications (the broker, publisher, and subscriber) are available at [[https://github.com/ccslab/CM/tree/emqtt/CM/src](https://github.com/ccslab/CM/tree/emqtt/CM/src)]([https://github.com/ccslab/CM/tree/emqtt/CM/src](https://github.com/ccslab/CM/tree/emqtt/CM/src)) (accessed on 14 November 2023). Through the experiments, we measured five performance criteria: end-to-end delay, publish-completion delay, broker-subscriber delay, packet sizes, and energy consumption. They are normal metrics to evaluate the performance of MQTT-related research works [56; 57; 58; 59]. Although we analyzed the number of exchanged messages in Section 3.1, the measurement of delay was necessary to figure out the overall overhead incurred by not only the number of messages but also other internal processing costs. Especially for delay, we measured delay values in different periods to analyze the overall cost from different point of views, as shown in Figures 1 and 3. The main purpose of measuring end-to-end delay ((1)-(8) in Figure 1 and (1)-(4) in Figure 3) was to measure total overhead in the end-to-end communication scenario that we addressed, and this metric was most important. We also measured publish-completion delay ((1)-(6) in Figure 1 and (1)-(5) in Figure 3) to compare the overhead of the normal message publication procedure because the E-MQTT broker has additional tasks when it receives PUBLISH and PUBREC packets. The aim of measuring broker-subscriber delay ((3)-(5) in Figure 1 and (2)-(3) in Figure 3) was to compare the response performance of the subscriber, looking for any side effects due to the procedure of E-MQTT. As well as the delay measurement, we also conducted the comparison of packet sizes because E-MQTT surely increases the size of some packets. Furthermore, we measured battery usage of the publisher to compare how much energy is consumed, out of regard for energy-constrained environments like IoT devices.  
Our test applications contained all the required codes to measure all the delay metrics and battery usage. To conduct the experiments, we configured the broker address in the publisher and subscriber applications. After starting the broker, we also started the publisher and subscriber. Then, they started MQTT connection with the broker and subscribed to topics required for the experiments. In the publisher application, we could select and start one of experiment menus (end-to-end delay, publish-completion delay, broker-subscriber delay, and battery usage). We could also select one of measurement methods among MQTT, asynchronous E-MQTT, and synchronous E-MQTT for comparison. For the delay measurement, the publisher sent a PUBLISH packet and measured the end-to-end and publish-completion delay values. The broker also measured the broker-subscriber delay value.