Package: paradigm-flood
🌊🌊 flood 🌊🌊floodis a load testing tool for benchmarking EVM nodes over RPCFor each RPC method,floodmeasures how load affects metrics such as:throughputlatency (mean, P50, P90, P95, P99, max)error ratefloodmakes it easy to compare the performance of:different node clients (e.g. geth vs erigon vs reth)different hardware/software configurations (e.g. local vs cloud vs low memory vs RAID-0)different RPC providers (e.g. Alchemy vs Quicknode vs Infura)floodcan generate tables, figures, and reports for easy sharing of results (example reporthere)ContentsInstallationPrerequisitesInstalling FloodDockerUsage GuideBasic Load TestsRemote Load TestsPrinting Test ResultsReport GenerationDifferential TestsFrom PythonPerforming Deep ChecksContributingInstallationPrerequisitesInstallvegeta:on mac:brew update && brew install vegetaon linux:go install github.com/tsenart/vegeta/v12@v12.8.4After installation, make surevegetais on your$PATH. Runningvegeta -hshould output a path. If it does not, you probably have not set upgoto install items to your$PATH. You may need to add something likeexport PATH=$PATH:~/go/bin/to your terminal config file (e.g.~/.profile).floodalso requirespython >= 3.7Installingfloodpip install paradigm-floodTypingflood helpin your terminal should show help output. If it does not, you probably have not set uppipto install items to your$PATH. You may need to add something likeexport PATH=$PATH:~/.local/binto your terminal config file (e.g.~/.profile). Alternatively, you can avoid setting up your$PATHand just typepython3 -m floodinstead offlood.DockerAlternatively, flood can be used as aDocker image.Usage guidefloodworks by bombarding an RPC endpoint with different patterns of RPC calls. Measurements of the RPC endpoint's performance under different controlled loads are then used to paint a detailed view of the node's performance.Every time flood runs, it saves its parameters and test results to an output directory. You can specify this output directory with the--outputparameter, otherwise a temporary directory will be created. Running a test will populate the folder with the following files:figures/: directory containing PNG's summarizing node performanceresults.json: results of the test including performance metricssummary.txt: printed summary of test that was output to the consoletest.json: metadata and parameters used to create and run the testBasic load testsHere is an example of a basic test withflood. It will benchmark block retrieval from two different nodes. It will test at 3 different rates (10, 100, and 1000 requests per second) and it will test them for 30 seconds each.flood eth_getBlockByNumber NODE1_NAME=NODE1_URL NODE2_NAME=NODE2_URL --rates 10 100 1000 --duration 30To see all of the parameters available for controllingfloodtests useflood --helpRemote load testsInstead of broadcasting RPC calls from whatever machine running thefloodCLI command,floodcan broadcast the calls from a remote process on a remote machine. In particular,floodcan broadcast the calls from the same machine that is running the EVM node in order to eliminate any noise or bottlenecks associated with networking.This can be accomplished by installing flood on the remote machine and then providingfloodwith login credentials and routing details using the following syntax:flood <test> [node_name=][username@]hostname:[test_url] ...For example, the following command will test a reth node and an erigon node remotely:flood eth_call reth=ubuntu@92.92.92.92:localhost:8545 erigon=ubuntu@91.91.91.91:localhost:8545If there are multiple remote tests, these tests will be run in parallel. After the tests are complete,floodwill retrieve the results and summarize using the same methodology as a local test.Printing test resultsBy defaultfloodproduces verbose output of each test as it runs. This can be disabled with the--quietparameter. To re-print the results of an old test, useflood print <TEST_DIR>. To print a summary of multiple tests, useflood print <test_1_dir> <test_2_dir>.Report generationAfter running tests, you can generate an HTML + Jupyter report similar to [this] one. This is done by runningflood report <TEST_DIR>. Multiple tests can be combined into one report withflood repos <TEST_DIR_1> <TEST_DIR_2> ....Differential testsInstead of testing the raw performance of an RPC node,floodcan be used to test the correctness of a node's responses using a differential testing approach. This works by using two nodes and making sure that their responses match under a variety of RPC calls. This is done using the--equalityparameter. For example:flood all reth=91.91.91.91 erigon=92.92.92.92 --equalityFrom pythonAll offlood's functionality can be used from python instead of the CLI. Some functions:descriptionpythonImport floodimport floodRun testsflood.run(...)Load the parameters of a testflood.load_single_run_results_payload(output_dir)Load the results of a testflood.load_single_run_results_payload(output_dir)Run a live version of a results notebookjupyter notebook <TEST_DIR>Performing deep checksUnder normal operationfloodrelies on vegeta to compute performance summaries of each test. This works well, but sometimes it is desirable to implement custom introspection not available invegeta.In particular,vegetacounts any status-200 response as a success, even if the contents of the response is an RPC error. Running with the--deep-checkcommand will check every response to make sure that it returns well-formed JSON with no RPC errors. With--deep-check,floodalso computes separate performance statistics successful vs failed calls.If you want to save the timing information and raw contents of every single response from the test to theresults.jsonoutput, use the--save-raw-outputargument. This allows for performing own custom analyses on the raw data.ContributingContributions are welcome in the form of issues, PR's, and commentary. Check out the contributor guide inCONTRIBUTING.md.