Package: pymerkle
pymerkleMerkle-tree in PythonDocumentation atpymerkle.readthedocs.org.Storage agnostic implementation capable of generating inclusion and consistency proofs.Installpip3installpymerkleThis will also installcachetoolsas a dependency.Basic APILetMerkleTreebe any class implementing theBaseMerkleTreeinterface; e.g.,frompymerkleimportInmemoryTreeasMerkleTreetree=MerkleTree(algorithm='sha256')Append data into the tree and retrieve the corresponding hash value:index=tree.append_entry(b'foo')# leaf indexvalue=tree.get_leaf(index)# leaf hashCurrent tree size:size=tree.get_size()# number of leavesCurrent and intermediate states:state=tree.get_state()# current root-hashstate=tree.get_state(5)# root-hash of size 5 subtreeInclusion proofProve inclusion of the 3-rd leaf hash in the subtree of size 5:proof=tree.prove_inclusion(3,5)Verify the proof against the base hash and the subtree root:frompymerkleimportverify_inclusionbase=tree.get_leaf(3)root=tree.get_state(5)verify_inclusion(base,root,proof)Consistency proofProve consistency between the states with size 3 and 5:proof=tree.prove_consistency(3,5)Verify the proof against the respective root-hashes:frompymerkleimportverify_consistencystate1=tree.get_state(3)state2=tree.get_state(5)verify_consistency(state1,state2,proof)Supported hash functionssha224,sha256,sha384,sha512,sha3_224,sha3_256,sha3_384,sha3_512Support for Keccak beyond SHA3Installingpysha3==1.0.2makes available
the following hash functions:keccak_224,keccak_256,keccak_384,keccak_512SecurityThis library requires security review.Resistance against second-preimage attackThis consists in the following standard technique:Upon computing the hash of a leaf node, prepend0x00to the payloadUpon computing the hash of an interior node, prepend0x01to the payloadNote: For, say, testing purposes, you can disable this feature by passingdisable_security=Truewhen initializing theBaseMerkleTreesuperclass.
Referhereto see how to perform second-preimage
attack against the present implementation.Resistance against CVE-2012-2459 DOSContrary to thebitcoinspecification, lonely leaves are not duplicated while the tree is growing.
Instead, a bifurcation node is created at the rightmost branch (see next section).
As a consequence, the present implementation should be invulnerable to theCVE-2012-2459DOS attack (see alsoherefor insight).TopologyInterior nodes are not assumed to be stored anywhere and no concrete links are
created between them. The tree structure is determined by the recursive
function which computes intermediate states on the fly and is essentially the same asRFC 9162(Section 2).
It turns out to be that of a binarySakura tree(Section 5.4).OptimizationsThe performance of a Merkle-tree depends on how efficiently it computes the root-hash
for arbitrary leaf ranges on the fly. The recursive version of this operation
(e.g.,RFC 9162, Section 2)
is slow.A key remark is that the above operation can be made iterative by combining the root-hashes
for ranges whose size is a power of two ("subroots") and can as such be computed
efficiently. Subroot computation has significant impact on performance
(>500% speedup) while keeping peak memory usage reasonably low
(e.g., 200 MiB for a tree with several tens of millions of entries) and
linear with respect to tree size.Note: For, say, comparison purposes, you can disable this feature by passingdisable_optimizations=Truewhen initializing theBaseMerkleTreesuperclass.CachingIn view of the above technique, subroot computation is the only massively repeated
and relatively costly operation. It thus makes sense to apply memoization
for ranges whose size exceeds a certain threshold (128 leaves by default).
For example, after sufficiently many cache hits (e.g. 2MiB cache memory), proof generation
becomes 5 times faster for a tree with several tens of million of entries.
Practically, a pretty big tree with sufficiently long uptime will respond instantly
with negligible penalty in memory usage.Note: For, say, comparison purposes, you can disable this feature by passingdisable_cache=Truewhen initializing theBaseMerkleTreesuperclass.StorageThis library is unopinionated on how leaves are appended to the tree, i.e., how
data is stored in concrete.  Cryptographic functionality is encapsulated in theBaseMerkleTreeabstract class, which admits pluggable storage backends
through subclassing. It is the the developer's choice to decide how to
store data by implementing the interior storage interface of this class.
Any contiguously indexed dataset should do the job. Conversely, given any such
dataset, we should be able to trivially implement a Merkle-tree that is
operable with it.ExampleThis is a simple non-persistent implementation utilizing a list as storage. It
expects entries to be strings, which it encodes in utf-8 before hashing.frompymerkleimportBaseMerkleTreeclassMerkleTree(BaseMerkleTree):def__init__(self,algorithm='sha256'):"""Storage setup and superclass initialization"""self.hashes=[]super().__init__(algorithm)def_encode_entry(self,data):"""Prepares data entry for hashing"""returndata.encode('utf-8')def_store_leaf(self,data,digest):"""Stores data hash in a new leaf and returns index"""self.hashes+=[digest]returnlen(self.hashes)def_get_leaf(self,index):"""Returns the hash stored by the leaf specified"""value=self.hashes[index-1]returnvaluedef_get_leaves(self,offset,width):"""Returns hashes corresponding to the specified leaf range"""values=self.hashes[offset:offset+width]returnvaluesdef_get_size(self):"""Returns the current number of leaves"""returnlen(self.hashes)DevelopmentIn what follows, you need to have locally installed dev requirements:pip3 install -r requirements-dev.txtTests./test.sh [--help]PerformanceIn order to capture the effect of I/O operations, performance measurements are
run against a SQLite database as leaf storage. Create it using the following script:python benchmarks/init_db.py [--help]Benchmarks./benchmark.sh [--help]ProfilingAssumingvalgrindandmassif-visualizerare installed, use./profile.sh [--help]to do memory profiling. Pass--timeto profile execution times
instead of memory allocations.Documentationpymerkle.readthedocs.org.Build locallyDocumentation is built withsphinx.Assuming dev requirements have been installed, build the docs with./build-docs.sh [--help]and browse atdocs/target/build/html/index.htmlto view them.