## 6 Evaluation and Discussion  
### Evaluation  
This part describes the evaluation results of the implementation of our proposed method. We tested our direct firmware update and peer-to-peer firmware update implementation. We measured the time taken when finishing some numbers of requests that were set up for testing. We also measured the average latency and the average throughput of both the Check New Version step and Download Update step. For this test, we utilized the Autocannon, which is an HTTP benchmarking tool [30]. It provides two ways of usage, with the command line or programmatically using Node.js programming language. It also offers several parameters that we could configure according to our test purposes. We evaluated our test in ten groups of requests; each group's total requests were increased by one hundred. It started with 100 requests, and the last group had 1000 requests. Table 4 shows the time performance for the Check New Version step. For the lowest number of requests, the direct firmware update protocol needs 4.28 s. To finish the last group, 1000 requests, this protocol needs 45.28 s. For peer-to-peer firmware update, the lowest number of requests, 100 requests, takes 5.22 s. The last group, 1000 requests, takes 36.26 s. Figure 10 shows the time performance for the Check New Version step of direct and peer-to-peer firmware update. The peer-to-peer firmware update protocol is faster than the direct firmware update protocol. \begin{table}
\begin{tabular}{c c c} \hline \hline \multirow{2}{*}{**Number of Requests**} & \multicolumn{2}{c}{**Duration (s)**} \\ \cline{2-3}  & **Direct Firmware Update** & **Peer-to-peer Firmware Update** \\ \hline
100 & 4.28 & 5.22 \\
200 & 9.26 & 8.30 \\
300 & 12.27 & 11.23 \\
400 & 20.22 & 16.32 \\
500 & 26.30 & 20.23 \\
600 & 28.24 & 21.33 \\
700 & 35.26 & 25.28 \\
800 & 36.24 & 29.26 \\
900 & 38.26 & 31.28 \\
1000 & 45.28 & 36.26 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Direct firmware update and peer-to-peer firmware update: the results of duration for the Check New Version step. Figure 9: Implementation of a peer-to-peer firmware update distribution. Figure 11 shows the comparison results of the average throughput and average latency performances of ten groups of requests in the Check New Version step for both protocols. Figure 11 shows that peer-to-peer firmware update protocol can handle 26.32 req/s. This value is higher than that of direct firmware update protocol. If there are 10,000 requests, the estimation time of peer-to-peer firmware update protocol to finish this step is 379.83 s or 6.3 min. The direct firmware update protocol can handle 22.1 req/s. That means that if there are 10,000 requests, the estimation time to finish this step is 452.38 s or 7.5 min. Figure 11 shows that the average latency of peer-to-peer firmware update is 360.92 ms. This value is lower than that for the direct firmware update protocol--437.52 ms. In terms of average latency and average throughput performance, the peer-to-peer protocol performs better than the direct update. The former communicates locally between gateways, while the latter communicates remotely between gateway and manufacturer. As the second part of the performance measurement, we measured the duration performance, the average latency, and the average throughput for the Download Update step. We evaluated with the same procedures as those for the Check New Version step. Table 5 shows the testing results of duration performance for the Download Update step of both direct firmware update and the peer-to-peer firmware update protocol. For the lowest request, the direct firmware update protocol needs 3.28 s, and for the last group, 1000 requests take 21.32 s. For peer-to-peer firmware updates, it takes 3.25 s to finish 100 requests, and it takes 19.23 s to finish 1000 requests. Figure 12 shows that the durations of both protocols to finish the second step are almost the same. However, the peer-to-peer protocol has a slightly lower duration than the direct firmware update protocol.