# An Integrated Platform for the Internet of Things Based on an Open Source Ecosystem
## 4 Evaluation
### System Performance Analysis  
[PERSON] et al. [82] proposed a comprehensive test methodology for IoT platform testing. The methodology includes functional, connectivity, performance, security, compatibility, and exploratory testing. It was applied to their SwarmOS platform, but only showed the HTTP response time with a different number of requests on three hardware platforms. They did not report the platform performance under different conditions. [PERSON], [PERSON] et al. [83] compared different web applications and IoT message protocol performances in a specific scenario. However, this test is not suited to industrial Internet of Things applications and does not include function testing. We compared the performance of different mainstream application layer transmission protocols under the condition of the industrial IoT and with the LoRA protocol used as the link layer protocol. Figure 7 shows the transmission performance test model of this system. Node-Red stands for the browser and runs the IoT application developed by Node-Red. The Eclipse Kura Gateway is the IoT gateway. The communication performance between the IoT application and the gateway is found by testing RTT (Round Time Live). \(\mathrm{T_{trans1}}\) is the request time, \(\mathrm{T_{process}}\) is the request processing time, \(\mathrm{T_{trans2}}\) is the response time of request, and \(\mathrm{T_{render}}\) refers to the time taken by the browser to display the received data. The performance tests were conducted according to Equations (1) and (2). The RTT performance of the CoAP and HTTP protocols were tested and compared using Jmeter and Jmeter-iot-lib based on Equation (1). The GET and PUT operations of the CoAP (POST for HTTP) protocol were compared and are shown in Figure 8 for the CON and NON-CON retransmission mechanism, respectively. \[\mathrm{RTT}=\mathrm{T_{trans1}}+\mathrm{T_{process}}+\mathrm{T_{trans2}} \tag{1}\]  
\[\mathrm{RTT}=\mathrm{T_{trans1}}+\mathrm{T_{process}}+\mathrm{T_{trans2}}+\mathrm{ T_{render}} \tag{2}\]  
In smart factories, some measurement data are small-sized, such as temperature, humidity, pressure, etc. However, in some cases--for example, when the network is congested--frequent data transmission may affect network performance, so the application can transmit the data measured multiple times at once. To check this situation, the transmission performance under different payloads was tested. The GET operation was used to obtain the resource data and the PUT operation was used to change the resource state. The CoAP block size was set to the default value. In Figure 8, it can be seen that the performance of the NON and CON mechanisms of the CoAP protocol was close when the  
Figure 7: RTT measurement. payload was small. When the payload was less than 1000 bytes, the RTT time was less than 10 ms in a wireless LAN environment, regardless of the CON or NON PUT and GET operations. When the transmission data were larger than the CoAP data block size, they increased faster because the data needed to be divided into smaller blocks to be transmitted several times. For example, when the payload was 1000 bytes, the RTT time of the NON CoAP was 10 ms; when the payload was 2000 bytes, the RTT time increased to 38 ms. When the payload was 3000 bytes, the RTT time was increased to 68 ms. The RTT time of the CON CoAP was 12, 53, and 76 ms, respectively. The overhead of CON mechanics also increased with the payload size. This indicates that, when the payload size is bigger than the size of the CoAP block size, the message will be transferred multiple times, and so the CON time is increased. In Figure 8, we show that the HTTP RTT time was twice that of NON CoAP when the payload size was less than 400 bytes. This is because the HTTP is connection-oriented and based on the TCP protocol. However, when the payload size is more than 1000 bytes, the RTT time of HTTP increased steadily. This is because of the different block size of the two protocols.