# Towards Blockchain-Based Federated Machine Learning: Smart Contract for Model Inference
## 4 Benchmarking Experiments
### Synthetic 2D Dataset Results  
The preparation step for the model validation use case experiment was uploading validation data to Hyperledger CouchDB through the chaincode. During the experiment, the peers uploaded their models. The uploaded models were validated using chaincode and the runtime of the validation function was measured. Blockchain was restarted and then the function that uses the oracle component rather than the chaincode was called, measuring the oracle service runtime. The median runtimes for all the tested network and data sizes are presented in Figure 8, where the bar plots indicate that the model inference time substantially increases after the dataset size reaches more than 8192 rows. Surprisingly, the linear increase in runtime with respect to the network size can only be noticed when testing on large datasets containing 16,348 or 32,768 rows. For the largest dataset scenario, we can observe a linear relationship between the network size and model inference runtime (see Figures 8 and 9) except for the outliers consistently resulting from the first run of the experiment. Interestingly, the outliers for the chaincode were fixed at almost a constant low value, while for the oracle service the outlier values fluctuated, especially when the network exceeded eight peers. The largest runtimes at the extreme scenario of large network and data settings are shown in Figure 10. We speculate that overheads are mainly due to the transferring of data from the blockchain state database to the oracle component and receiving back the model output, required for peers to arrive at a consensus to confirm the transaction. Most of the runtime results are clustered around 1 min and 19 s for both model inference implementations. Figure 10 also displays that both types of validation methods displayed the lowest runtime values during benchmarking initialization runs, as the transaction count between peers were still low and peers were not busy computing new blockchain blocks. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Data Records** & **Synthetic Dataset (kB)** & **EEG Eye State Dataset (kB)** \\ \hline
1024 & 114 & 510 \\
2048 & 227 & 1023 \\
4096 & 453 & 2049 \\
8192 & 905 & 4097 \\
16,348 & 1809 & 14,446 \\
32,768 & 3650 & 16,384 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Amount of data records and size (in kB) of the corresponding JSON (JavaScript Object Notation) structure. Table 3 demonstrates the runtime overhead in percentages of the oracle service component. Results indicate that the synthetic dataset, surprisingly, had both negative and positive overheads with an overall median overhead of 1.99%. The results could be summarized as follows: (a) On all network sizes (3-13 peers) and data records not exceeding 8192 (1024-8192 records) the overhead ranges from \(-\)4.31 to 6.59%. As validation with these configurations takes a relatively short time, the variability in inference overhead is quite high; (b) as data increase to 16,348 records variability in overhead drops for all peers (3-13 peers) with the overhead ranging from 0.73 to 4.28%; (c) using the highest amount of data records tested (32,768 records), the variability in validation times drops significantly only producing at most 2.79% overhead on the least number of peers (three peers) tested and less than 1% (0.03-0.73%) on the remaining peer configurations (5-13 peers). Results indicate that the overhead diminishes once the number of participants and data records increase. The higher variability of overheads for smaller dataset sizes irrespective of the network configuration was observed. Figure 8: Model inference runtime (in seconds) depending on the network size for different amounts of the synthetic dataset. Figure 9: Model inference runtime (in seconds) in relation to the network size for the largest variant of the synthetic dataset: Chaincode (**a**) and oracle service (**b**) results.