## 5 Evaluation
### Performance Benchmark
#### 5.3.1 Publication Rate as a Variable  
Four experiments were conducted in this category. All variables are fixed, except the publication rates of the information provider application, which were varied between one and 40 publications per second, and all publication and notification messages contain only one parameter. The first experiment applied the security measures (_i.e._, HTTPS and HTTP basic authentication) in a standalone IIP set-up, while the second experiment did not incorporate any security mechanism, also in a standalone set-up. The third and fourth experiments were conducted in a clustered IIP set-up, where security was applied  
Figure 17: Snapshot of ndb_mgm client showing the most current MySQL Cluster configuration: (**a**) Both data and SQL node hosts are up and running; (**b**) One data and SQL node host (192.168.1.7) are down. in the third experiment and was ignored in the fourth experiment. Each measurement in all experiments lasted five minutes, generating 300 to 12,000 data messages (depending on the publication rate). Each measurement was performed three times to ensure data consistency, and the first 1% of the captured data is removed from every measurement to avoid the start-up effect of the application servers in serving incoming requests. All measurement data were averaged and plotted alongside confidence intervals at the 95% confidence level. Figure 18a shows a comparison of total average publication and notification times between the secured and non-secured standalone IIP set-ups. It can be seen that the average difference at one publication per second is about 85 ms, which is the rough estimate of the security scheme's overhead in the standalone set-up. This average difference does not change much with the increase of the publication rate until around 30 publications per second. From there, the difference gap grows larger significantly. Figure 18b shows a comparison of similar latency measurements as in Figure 18a, except that the compared experimental results are between secured and non-secured clustered versions of the IIP. In a clustered set-up, the starting difference between secured and non-secured implementations is about  
Figure 18: Publication and notification time comparisons with publication rate as variable: (**a**) Standalone secure _vs_. insecure; (**b**) Clustered secure _vs_. insecure; (**c**) Secure standalone _vs_. clustered; (**d**) Insecure standalone _vs_. clustered. 90 ms, slightly higher than in the standalone set-up. This difference gap is maintained in a relatively stable manner up to 40 publications per second. From Figure 18a,b, it can be seen that the clustered version of the IIP can handle the increase of the publication rate better than its standalone counterpart, especially with the security scheme being applied. A latency comparison between secured standalone and clustered IIP set-ups is depicted in Figure 18c. The average difference gap is stabled at about 20 ms up to around 25 publications per second, and then, it grows more than two times. This can be viewed as the lesser ability of the standalone IIP set-up in handling faster publication rates compared to the clustered set-up. Figure 18d shows a latency comparison for publication and notification between non-secured standalone and clustered IIP set-ups. The average difference stays almost unchanged at about 20 ms from one to 25 publications per second, and the gap slightly widens as the publication rate increases. From the four experiments conducted in this category (_i.e._, publication rate as a variable), it can be concluded that the clustered deployment of the IIP handles higher rates of publications better than the standalone set-up. Even at lower rates, the total average latency for publication and notification are smaller in the clustered set-up, although only by a small margin. The inclusion of the security scheme adds additional overhead to the overall processing times in both standalone and clustered set-ups.