DOI: 10.5281/zenodo.17672003
title: Multiple Blockchain Consensus Performance Evaluation

description: NANCY project has received funding from the Smart Networks and Services Joint Undertaking (SNS JU) under the European Union’s Horizon Europe research and innovation programme under Grant Agreement No 101096456. 

Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of theEuropean Union or the SNS JU. Neither the European Union nor the granting authority can be held responsible for them.


Environment setup

The data creation process includes a combination of internally generated experimental data and simulated workloads designed to emulate real-world blockchain environments. The primary focus was to analyse the latency and throughput characteristics of various blockchain systems and consensus protocols, specifically the ones mentioned in Table1:

Table 1: Systems Under Test (SUTs)




Platform
Version
Consensus Protocol


Hyperledger Fabric
v2.5
Raft (CFT)


Hyperledger Fabric
v3.0
SBFT


Hyperledger Besu
v25.0
QBFT, IBFT2.0




These decentralised networks were deployed in a controlled environment that allowed to monitor, record, and extract key performance metrics under different configurations and workloads. The data collection focused on network latency, block propagation times, transaction finality and overall system responsiveness under various stress levels. The experiment was conducted on blockchain solutions that are appropriate for industrial use cases. The tool that was used for conducting the tests was Hyperledger Caliper [7], an open-source benchmarking tool designed to evaluate the performance of blockchain networks. It allows users to run predefined or custom test workloads on various blockchain platforms and generates detailed reports on key performance metrics such as transaction latency, throughput (TPS), and success/failure rates.

The characteristics of the workstation that was used for the benchmarking process is described in Table 2.

Table 2: Workstation specifications




Component
Specification




Operating System 

Ubuntu 24.04.2 LTS




Motherboard

Gigabyte Z790 AORUS ELITE AX




CPU

Intel® Core™ i7-14700KF (20-core: 8 P-cores + 12 E-cores, up to 5.5 GHz)




Memory

64 GiB DDR5 RAM (2×32 GiB @ 5600 MHz)




Benchmark configuration

Figure1 is an example of a benchmark configuration file that was used for the tests.

 

Figure 1: Benchmark config file

Each key parameter corresponds to a specific value parameter, with the key - value pairs explained Table 3.

Table 3: Benchmark configuration explanation




Parameter
Description


test.workers
Configuration for how transaction load is generated


test.workers.number
Specifies the number of worker processes to use for executing the workload


test.rounds
Array of objects, each describing the settings of a round


test.rounds[i].label
A short name of the rounds, usually corresponding to the types of submitted TX


test.rounds[i].txDuration
The length of the round in seconds during which Caliper will submit TXs


test.rounds[i].rateControl
The object describing the rate controller to use for the round


test.rounds[].rateControl.type
The rate control strategy. fixed-rate means transactions are sent at a steady, constant rate.


test.rounds[].rateControl.opts.tps
TPS to send. Here, Caliper will attempt to submit 100 transactions per second throughout the 60- second test


test.rounds[i].workload
The object describing the workload module used for the round


test.rounds[i].workload.module
The path to the benchmark workload module implementation that will construct the TXs to submit




A configuration with the above structure will define a benchmark run that consists of multiple rounds. Each round is associated with a rate controller that is responsible for the scheduling of transactions (TXs) and a workload module that will generate the actual content of the scheduled TXs.

The benchmarking included various transaction rates, ranging from 5 to 452 TPS in fixed steps (Table 4). Each test round lasted 500 transactions, with 1 to 10 local worker threads generating load. The same application workload was used across all tests. The standard FabCar smart contract was used, simulating realistic asset creation scenarios. The controller of choice was fixed rate to simulate a stable transaction load and to mimic a high but stable demand of transaction executions. The fixed-rate controller sends transactions at a constant, pre-defined rate (i.e., a fixed number of TPS).

Table 4: Benchmark round parameters




Parameter
Description


Benchmarking tool
Hyperledger Calieper v0.6


Transaction Rates (TPS)
5 to 452 (in fixed steps of 3)


Test duration
500 transactions per round


Load Generation
1-10 local workers


Workload
createCar and queryCar transaction for the FabCar smart contrac


Network configuration
Varying number of orderers/validators depending on protocol




Workload generation setup

A sample of the workload module used for the testing is provided in Figure 2

 

Figure 2: Workload module

Workload modules are implemented as Node.JS [8] modules that expose a certain Application Programming Interface (API). There are no further restrictions on the implementation, allowing developers to implement arbitrary logic (using further arbitrary components).

These modules are loaded through factory functions, just like other pluggable modules in Hyperledger Caliper. Accordingly, a workload module implementation must export a single factory function, named “createWorkloadModule”. The factory function must return an instance that implements the [WorkloadModuleInterface]. The “submitTransaction” function is the backbone of the workload generation. The worker process calls this function every time the rate controller enables the next TX and is its responsibility to submit the TX through the connector API.

Key Performance Metrics

Hyperledger Caliper provides a detailed set of performance metrics after each benchmarking test, helping you evaluate how well a blockchain system performs under load. Table 8 summarizes the key metrics Caliper returns and their corresponding descriptions.

Table 8: Network performance metrics




Metric
Description


Success
Number of transactions that were successfully submitted and confirmed on the blockchai


Failed
Number of transactions that were submitted but failed due to errors like timeouts, endorsement issues, or network problem


Send Rate (TPS)
The average number of transactions sent per second during the test.


Throughput (TPS)
The number of successful transactions per second (i.e., how many were confirmed on-chain).


Latency (min/avg/max)
Time taken for a transaction to go from submission to confirmation. Caliper reports the minimum, average, and maximum latency values.




References:

[1] “Hyperledger Fabric.“ [Online]

[2] “The Raft Consensus Algorithm.” [Online]

[3] Y. Manevich, Y. Tock, and H. Meir, “Hyperledger Fabric v3: Delivering Smart Byzantine Fault Tolerant consensus” Sep. 2024. [Online]

[4] “Besu” [Online]

[5] “Configure QBFT consensus” Apr. 2025. [Online]

[6] “Configure IBFT 2.0 consensus” Apr 2025. [Online]

[7] “Caliper” [Online]

[8] “Node.js - Run JavaScript Everywhere” [Online]