Package: treesimi
treesimi: Shingling for measuring tree similarityCompute similarity between trees, e.g. dependency treesConvert an Adjacency List into a Nested Set TableFor example, CoNLL-U's['id', 'head']fields form an adjacency list of a dependency tree.
Traversing an adjacency list is slower than reading a nested set.
Thus, converting a adjacency list to a nested set table once, makes sense if we need to read the three several times lateron.importtreesimiastsadjac=[(1,0),(2,1),(3,1),(4,2)]nested=ts.adjac_to_nested(adjac)# columns: node id, left, right, depth# [[1, 1, 8, 0], [2, 2, 5, 1], [4, 3, 4, 2], [3, 6, 7, 1]]Demo: Query a nested set tableTo extract a subtree we just need to iterate through the list ($O(n)$)_,lft0,rgt0,_=nested[1]subtree=[(i,l,r,d)fori,l,r,dinnestedif(l>=lft0)and(r<=rgt0)]# [[2, 2, 5, 1], [4, 3, 4, 2]]orts.get_subtree(nested, sub_id=2)Set node attributesimporttreesimiastsnested=[[1,1,8,0],[2,2,5,1],[4,3,4,2],[3,6,7,1]]attrs=[(1,'a'),(2,'b'),(3,'c'),(4,'d')]nested=ts.set_attr(nested,attrs)# columns: node id, left, right, depth, attributes# [[1, 1, 8, 0, 'a'], [2, 2, 5, 1, 'b'], [4, 3, 4, 2, 'd'], [3, 6, 7, 1, 'c']]Convert Adjacency List with attributesimporttreesimiastsadjac=[(1,0,'dat1'),(2,1,'dat2'),(3,1,'dat3'),(4,2,'dat3')]nested=ts.adjac_to_nested_with_attr(adjac)# columns: node id, left, right, depth# [[1, 1, 8, 0, 'dat1'], [2, 2, 5, 1, 'dat2'], [4, 3, 4, 2, 'dat2'], [3, 6, 7, 1, 'dat4']]Extract subtree patternsWe can extract the following patterns from one tree:Depth dimensionFull subtreesTruncate leavesSibling dimensionAll siblingsDrop siblings (and their subtree)Placeholder attribute fieldFull subtreesThe functionextract_subtreesreturns all subtrees of a tree.
The depth information is adjusted accordingly for each subtree.importtreesimiastsnested=[[1,1,8,0,'a'],[2,2,5,1,'b'],[4,3,4,2,'d'],[3,6,7,1,'c']]nested=ts.remove_node_ids(nested)subtrees=ts.extract_subtrees(nested)# [#    [[1, 8, 0, 'a'], [2, 5, 1, 'b'], [3, 4, 2, 'd'], [6, 7, 1, 'c']],#    [[1, 4, 0, 'b'], [2, 3, 1, 'd']],#    [[1, 2, 0, 'd']],#    [[1, 2, 0, 'c']]# ]Truncate leavesIn the first step, the functiontrunc_leavesremoves leaves of the largest depth level.
The result is always an incomplete tree, and thelftandrgtvalues arenot adjustedto indicate thatthere is a missing node.
In the next steps, the depth level is further removed down todepth=1.importtreesimiastsnested=[[1,1,8,0,'a'],[2,2,5,1,'b'],[4,3,4,2,'d'],[3,6,7,1,'c']]nested=ts.remove_node_ids(nested)subtrees=ts.trunc_leaves(nested)# [#   [[1, 8, 0, 'a'], [2, 5, 1, 'b'], [6, 7, 1, 'c']]# ]Hint: Runtrunc_leavesfor each subtree extracted byextract_subtrees. Callunique_treesafter each step.Drop sibling nodesGenerate variants of a tree by dropping each node once.
Again, the result is always an incomplete tree, and thelftandrgtvalues arenot adjustedto indicate thatthere is a missing node.importtreesimiastsnested=[[1,1,8,0,'a'],[2,2,5,1,'b'],[4,3,4,2,'d'],[3,6,7,1,'c']]nested=ts.remove_node_ids(nested)subtrees=ts.drop_nodes(nested)# [#   [[1, 8, 0, 'a']],#   [[1, 8, 0, 'a'], [2, 5, 1, 'b']],#   [[1, 8, 0, 'a']]# ]Hints: Create subtrees withextract_subtreesandtrunc_leaves, and rundrop_nodeson these subtrees. If you want to drop N nodes/leaves of a tree, then call the function twice, e.g.drop_nodes(drop_nodes(...)).Placeholder attribute fieldThereplace_attrremoves the data attribute of a node with a generic placeholder.importtreesimiastsnested=[[1,1,8,0,'a'],[2,2,5,1,'b'],[4,3,4,2,'d'],[3,6,7,1,'c']]nested=ts.remove_node_ids(nested)subtrees=ts.replace_attr(nested,placeholder='[MASK]')# [#   [[1, 8, 0, '[MASK]'], [2, 5, 1, 'b'], [3, 4, 2, 'd'], [6, 7, 1, 'c']],#   [[1, 8, 0, 'a'], [2, 5, 1, '[MASK]'], [3, 4, 2, 'd'], [6, 7, 1, 'c']],#   [[1, 8, 0, 'a'], [2, 5, 1, 'b'], [3, 4, 2, '[MASK]'], [6, 7, 1, 'c']],#   [[1, 8, 0, 'a'], [2, 5, 1, 'b'], [3, 4, 2, 'd'], [6, 7, 1, '[MASK]']]# ]Demo Notebooks about Shingling for MinHashWe recommend using themmh3hash function, and 32 permutations indatasketch.MinHash.Create subtrees as shingle setsJaccard Similarity between Dependency TreesShingle Dependency Trees for datasketch's MinhashStart jupyter to run the demo notebooksource.venv/bin/activate
jupyterlabAppendixInstallationThetreesimigit repois available asPyPi packagepipinstalltreesimi
pipinstallgit+ssh://git@github.com/ulf1/treesimi.gitCommandsInstall a virtual environmentpython3-mvenv.venvsource.venv/bin/activate
pipinstall--upgradepip
pipinstall-rrequirements.txt--no-cache-dir
pipinstall-rrequirements-dev.txt--no-cache-dir
pipinstall-rrequirements-demo.txt--no-cache-dir(If your git repo is stored in a folder with whitespaces, then don't use the subfolder.venv. Use an absolute path without whitespaces.)Python commandsCheck syntax:flake8 --ignore=F401 --exclude=$(grep -v '^#' .gitignore | xargs | sed -e 's/ /,/g')Run Unit Tests:pytestPublishpythonsetup.pysdisttwineupload-rpypidist/*Clean upfind.-typef-name"*.pyc"|xargsrm
find.-typed-name"__pycache__"|xargsrm-r
rm-r.pytest_cache
rm-r.venvSupportPleaseopen an issuefor support.ContributingPlease contribute usingGithub Flow. Create a branch, add commits, andopen a pull request.AcknowledgementsThe "Evidence" project was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) -433249742(GU 798/27-1; GE 1119/11-1).Maintenancetill 31.Aug.2023 (v0.2.0) the code repository was maintained within the DFG project433249742since 01.Sep.2023 (v0.3.0) the code repository is maintained by Ulf Hamster.