Package: atomkraft
Atomkraft: E2E testing for Cosmos blockchainsBelow we describe what Atomkraft is about, and explain the concepts behind the tool. In case you would like to skip that, and jump directly into action, please read ourInstallation guideto install the tool; afterwards it's worth following either ourCosmos SDK Token Transfer tutorial, orCosmWasm tutorial.We cover the following topics in this file:Atomkraft IntroductionConceptual overviewAtomkraft projectLocal testnetsTraces and reactorsGenerating traces from TLA+ modelsRunning the tests against testnetWhat's next: Atomkraft's immediate futureIntroductory videoAtomkraft IntroductionTheCosmos NetworkofIBC-connected blockchains is growing tremendously fast. There is one aspect though, which has not been fully addressed yet, namelyquality assurance: how do we make sure that Cosmos-based blockchains are secure, and don't contain security issues that pose hazards for user funds?Atomkraftis our answer to that question: it allows to generate and execute massive end-to-end (E2E) test suites for Cosmos SDK based blockchains. When designing the tool, we keep two main categories of users for it:security auditors, andCosmos SDK blockchain developers. Those roles are abstract and non-exclusive, and they serve only to summarize some important concerns wrt. the deployed blockchains. Is Atomkraft the right tool for you? Below are some possible hints.Atomkraft is likely to benefit you as aSecurity auditorif:your blockchain (or the new version of it) is about to be launched, but you are unsure whether incentivized testnets allowed you to discover all critical vulnerabilities;you are on tight time schedule for the new release, and you feel that the implementation has been rushed a little, so you are unsure that all corner cases of the new functionality has been covered.Atomkraft is likely to benefit you as aCosmos SDK developerif:the new functionality you've implemented interacts in non-trivial ways with other important modules (e.g. withbank,staking,authz, etc.), and you are unsure whether some important invariants of those modules are preserved;you want to grow and maintain a regression test suite for the blockchain modules you are developing, to make sure of their correctness as your blockchain evolves;you want to automate quality assurance for your blockchain modules, and integrate fast E2E testing solution that's executed on every PR.Key Atomkraft featuresthat allow to address the above concerns:Push-button, but fully customizable, automation of local testnet creationClean, fully customizable trajectory from test case generation to its execution on the local testnetSupport for automatic generation of massive test suites from compact TLA+ modelsEasy execution of test cases generated via other means (e.g. manual, BDD, PBT)Anytime ready-to-integrate regression test suite in the form of a standard Pytest projectReady-to-execute standard test suites for important Cosmos SDK modules (coming soon!)Generation of reports and dashboards for presentation and analysis of testing results (coming soon!)Conceptual overviewIn itself, Atomkraft is nothing more than a command-line application, which is as easy to obtain for your system as executingpip install atomkraft; please consult the detailedInstallation Instructionsif needed. At the top-level, Atomkraft provides you the commands illustrated in the diagram below.Atomkraft projectWorking with Atomkraft takes place in the scope of anAtomkraft project, which is initialized viaatomkraft initcommand. The command will setup a new project in a given directory and install all necessary software for it, all fully automatically.The good news is that an Atomkraft project is simultaneously aPytestproject! At any point in time you can executepytestinside your Atomkraft project, and this command will rerun all tests you've created in the scope of this project. So, when you've executedatomkraft init, you've also started to create your regression test suite; congratulations!More than that, an Atomkraft project is also aPython Poetryproject: you can treat it just as a normal Python project, and add your custom Python tests there.At the top level, an Atomkraft project contains the following:Folders that hold the corresponding artifacts:modelsis meant to keep your TLA+ models, if you are willing to generate test cases from formal models;tracescontain abstract test cases, represented in the form ofITF traces(a JSON encoding of a sequence of test case steps);reactorshold Python functions that interpret ITF traces, and map trace steps to concrete blockchain transactions;intestswe collect all reproducible Pytest tests that are executed in the scope of this Atomkraft project;finally,reportshold the test reports.Configuration files:atomkraft.tomlcontains Atomkraft project configuration;chain.tomlcontains the blockchain configuration parameters, such as the number of nodes or the number of validators;model.tomlcontains the TLA+ model configuration parameters, used to run model checkers;pyproject.tomlcontains the Poetry project configuration.All of the above configuration files are filled with the default configuration, so you don't need to worry about them, if you don't want to. On the other hand, when needed, everything is fully customizable.Local testnetsWith Atomkraft project created, you should be ready to go. By default, we configure local testnets to use thegaiad(Cosmos Hub) binary. If it is available via yourPATH, executingatomkraft chain testnetshould bring up a local testnet with 2 nodes and 3 validators. If you would like to configure any parameters differently (e.g. to run your custom blockchain binary), you can do it either viaatomkraft chain configcommand, or by directly editingchain.tomlconfig file. Please make sure your changes are valid by executingatomkraft chain testnet; we use the local testnet to run the tests.Traces and reactorsFor describing test cases we use theInformal Trace Format, which is a JSON encoding of a sequence of steps, and each step encodes values of key state variables; please seethis example trace. The trace has been produced by our in-houseApalache model checkerfromthis TLA+ model.ITF traces are abstract; in the example trace above, it holds wallet balances as a simple mappingbalancesfrom abstract addresses, represented as integers, to their balances, also represented as integers without denomination. There are two abstract actions in this trace:Init, andTransfer. In order to be able to replay the abstract trace on a blockchain, each of those abstract actions needs to be translated to a concrete transaction, as well as all abstract parameters of an action need to be translated to concrete values (e.g. an abstract integer address needs to be translated into a concrete Cosmos address). This translation step is performed by a component that we callreactor: a reactor is a centerpiece for an Atomkraft project, without which it can't function, similar to a nuclear reactor for the atomic power plant. You can seethis example reactorthat is able to playback the above trace on the blockchain.This separation between abstract traces and reactors, which apply abstract traces to concrete blockchains, is the crucial aspect of Atomkraft infrastructure; here is why:Abstract traces are more maintainable than concrete tests. Concrete tests need to be updated on every small change in encoding, API, etc.; we have seen PRs with thousands of LOC concerning only with updating manually written tests. With the separation into traces and reactors, traces won't need to be updated in most cases; only the relatively small reactors will need small tweaks.Abstract traces can be generated by whatever means: from model checkers such asApalache, via fuzzing, PBT, BDD, or even manually. How the abstract trace is produced, doesn't matter; it still can be executed using Atomkraft's infrastructure.It is much easier to understand the intent of a test case expressed as an abstract trace, because of absence of excessive details.Abstract traces are lightweight in terms of storage, which allows us to generate and maintain thousands of them, covering many corner cases, at no extra cost.We have automated the process of writing a reactor viaatomkraft reactorcommand. A user needs only to supply the lists of actions, and of state variables, and the command will generate a reactor stub with a function for each action; what remains is only to fill the body of each such function.Generating traces from TLA+ modelsAs explained above, abstract traces can be obtained by whatever means; we do not constrain the user in this respect. The most time efficient method, from our point of view, is to generate traces from formal models expressed inTLA+, the specification language designed by Leslie Lamport. For a gentle introduction to TLA+ you may use the Informal'sTLA+ Language Reference ManualandTLA+ Basics Tutorial. While TLA+ may look scary for beginners, we can assure you that learning it will greatly improve your productivity when reasoning about (and testing!) both protocols and code.The good news is that we have done a thorough work in making user's life as easy as possible when working with TLA+ models, and using them to generate abstract test traces. All heavy-lifting work wrt. TLA+ models is done by ourApalachemodel checker. The model checker itself is meant for expert users; Atomkraft tries its best to hide excessive complexity from its users and exposes only the most essential functionality for working with models. You can access this functionality viaatomkraft modelcommand, which provides you with the functions like listed in the screenshot below:The most important command in the scope of test case generation isatomkraft model sample. E.g. the command below (assuming you are in the Atomkraft project)atomkraftmodelsample--model-pathmodels/transfer.tla--traces-dirtraces--testsExwill generate an abstract trace from thetransfer.tlamodel and the test predicateEx, and store the generated trace in thetracesdirectory of your Atomkraft project.Running the tests against testnetLet's assume you've done all the steps outlined above:created a fresh Atomkraft project usingatomkraft initconfigured a Cosmos-based blockchain of your choice usingatomkraft chaincreated a reactor stub usingatomkraft reactor, and populated it with codegenerated abstract traces from a TLA+ model usingatomkraft model sample, or created abstract traces by some other meansThen you are ready to go, and execute your tests! We provide two commands for doing that:atomkraft test tracewill accept an abstract trace and a reactor, spin the local testnet, and execute the trace against the testnetatomkraft test modelis a convenience shorthand that combines step 4 above (atomkraft model sample) withatomkraft test trace, and allows you to execute tests directly from a TLA+ model, sidestepping explicit trace generation.Both of the aboveatomkraft testcommands populate thetestsdirectory of your project with Pytest-based tests; so executingpytestinside your Atomkraft project at any point in time will reproduce all of your tests. In fact, the complete Atomkraft project directory is ready at any point in time to be exported, and used as a Pytest project, for example for reproducing your tests in the CI.What's next: Atomkraft's immediate futureAtomkraft's functionality outlined above represents the tool MVP: please feel free to employ it in your projects, and let us know of your experience: we are always ready to assist!There are many more features that are planned or are already being implemented; so stay tuned. Below is the preview of the future Atomkraft functionality:Standard test suites: we have already started the effort to provide standard reactors and tests for most important Cosmos SDK modules:bank,staking,authz; you name it! This will serve the community at large, and will allow you, as an Atomkraft user, to easily bootstrap your new Atomkraft projects:running standard test suites in your CI will make sure that your new functionality doesn't break the important blockchain invariants;using standard test suites as blueprints will allow you to easily create your own tests suites via examination and adaptation of already existing tests.Test understanding and debugging: we started to work in the direction of assisting the user in simplified creation and understanding of the test scenarios, as well as debugging the failed test runs:for the former, we will provide the differential trace viewer, which highlights only the changes between the trace steps;for the latter, we will provide the built-in blockchain explorer, and integrate into it the capabilities to trace back and forth between abstract trace steps and concrete blockchain transactions.Test reports and dashboards: we plan to implement the functionality for generation of test reports as well as live dashboards that would provide an easy-to-grasp overview and categorization of executed and running tests.Exhaustiveness: we plan to implement certain coverage metrics (e.g. transaction sequences up to specified length), and to help user achieving full coverage according to those metrics to provide confidence in the code in case no bugs are discovered.Introductory videoWe have recorded a short video illustrating the basic Atomkraft usage: