Package: pathtreelib
Getting startedThe main class is PathTree and provides 5 different categories of
functionalities:Generate the structure of PathNodes (at tree initialization)Different types of iteration on the nodes (e.g. depth-first, breadth-first)Computation of different types of custom properties (e.g. bottom-up, top-down)Pruning of part of the treesExporting in csv and ExcelFor the sake of the following examples, suppose the directories are organized as
follow.Moreover, the analytics subpackage provide the class PathTreeAnalytics, a
subclass of PathTree, that contains few additional methods that perform more
advanced analysis on the tree. Few usage examples of such methods are provided
below.Generate the treeThe initialization of the tree generate all the nodes and requires only the
root.>>> from pathtreelib import PathTree
>>> tree = PathTree("gordon")Node iteratorsClassic breadth-first and depth-first iterators are implemented. Breadth-first
is used as default iterator in __iter__.>>> for node in tree.breadth_first_iter():
>>>     print(node.path.as_posix())
    gordon
    gordon/black mesa
    gordon/pictures
    gordon/black mesa/research.pdf
    gordon/black mesa/xen.txt
    gordon/pictures/equipment
    gordon/pictures/holiday
    gordon/pictures/equipment/crowbar.jpg
    gordon/pictures/equipment/gravitygun.png
>>> for node in tree.depth_first_iter():
>>>     print(node.path.as_posix())
    gordon
    gordon/black mesa
    gordon/black mesa/research.pdf
    gordon/black mesa/xen.txt
    gordon/pictures
    gordon/pictures/equipment
    gordon/pictures/equipment/crowbar.jpg
    gordon/pictures/equipment/gravitygun.png
    gordon/pictures/holidayAn additional iterator, based on breadth-first, allows to skip subtrees that
does not satisfy a specific condition.>>> valid_func = lambda node: node.path.property[PathTreeProperty.DEPTH] < 3
>>> for node in tree.validated_iter(valid_func):
>>>     print(node.path.as_posix())
    gordon
    gordon/black mesa
    gordon/black mesa/research.pdf
    gordon/black mesa/xen.txt
    gordon/pictures
    gordon/pictures/equipment
    gordon/pictures/holidayProperty computationThe supported properties are classified by the type of algorithm used for the
computation. Currently the library supports three types of properties: bottom-up
properties, top-down properties, and individual properties. The properties are
computed on all the nodes and require computation functions.Bottom-up propertiesBottom-up properties are computed on the children first and then on the parent,
hence are computed from the leaves upward. An example of this kind of properties
is the height.The computation requires the function to compute the property on a leaf>>> def leaf_func(leaf:PathNode) -> int:
>>>     return 0and the function to compute the property on an inode, assuming it is already
computed on its children,>>> def inode_func(inode:PathNode, children:list[PathNode]) -> int:
>>>    children_min_height = min([child.property["height"] for child in children])
>>>    return children_min_height + 1then the property computation function ensures that the property is computed on
the nodes in the correct order.>>> tree.compute_bottom_up_property("height", leaf_func, inode_func)
>>> print(tree.root.property["height"])
    2Top-down propertiesTop-down properties are computed on the parent first and then on the children,
hence are computed from the root downward. An example of this kind of property
is the depth.The computation requires the function to compute the property on the root>>> def root_func(root:PathNode) -> int:
>>>     return 0and the function to compute the property on all other nodes, assuming it is
already computed on its parent,>>> def notroot_func(node:PathNode, parent:PathNode) -> int:
>>>     return 1 + parent.property["depth"]then the property computation function ensures that the property is computed on
the nodes in the correct order.>>> tree.compute_top_down_property("depth", root_func, notroot_func)
>>> print(tree.root.property["depth"])
    0Individual propertiesIndividual property are computed on a node independently from the others, hence
the order of the computation is irrelevant. An example of this kind of property
is a flag indicating if the node’s path is a directory.The computation requires the function to compute the property on a generic node>>> def property_func(node) -> bool:
>>>     return node.path.is_dir()then the property computation function applies the property on all the nodes.>>> tree.compute_individual_property("is_dir", property_func)
>>> print(tree.root.property["is_dir"])
    TrueProperty removalThe properties can also be removed from all nodes of the tree.>>> tree.remove_property("height")PruningThe pruning allows to reduce the nodes in the tree by removing subtrees that do
not satisfy a specific condition. The removal of the nodes can be physical,
links removed from the nodes, or logical, properties added to the node.An example of condition is a limit on the depth.>>> keep_condition = lambda node: node.property[PathTreeProperty.DEPTH] <= 2The logical pruning add the pruning propery but does not change the structure of
the tree and is reversable.>>> tree.logical_pruning(keep_condition=keep_condition)
>>> print(len(list(node for node in tree)))
    9
>>> print(len(list(node for node in tree if not node.property[PathTreeProperty.PRUNED])))
    7The physical pruning removes the links between nodes, hence the structure of the
tree changes irreversibely.>>> tree.physical_pruning(keep_condition=keep_condition)
>>> print(len(list(node for node in tree)))
    7ExportThe tree can be exported in csv and Excel format allowing a set of
customization. With basic setting, the export includes the path and the
properties, however it is possible to filter on the properties and an the nodes.>>> tree.to_csv(csvfile="test.csv", properties["heigth", "depth", "is_dir"])
>>> tree.to_excel(csvfile="test.xlsx", properties["heigth", "depth", "is_dir"])ExampleAnalysis of the whole C volume in Windows to find the most space consuming
folders with holiday pictures (folders with a large number of jpg files).# Generate the whole tree (could take a while if the volume is large)tree=PathTree("C:/")# Compute on each node the number of picture in its subtreetree.compute_bottom_up_property("num_of_jpg",leaf_func=lambdaleaf:1ifleaf.path.suffix=='.jpg'else0,inode_func=lambdainode,children:sum([child.property["num_of_jpg"]forchildinchildren]))# Keep only the subtrees of interest (that have at least 10 pics)tree.physical_pruning(keep_condition=lambdanode:node.property["num_of_jpg"]>10)# The nodes that are now leaves are the photo directories we are looking fortree.compute_individual_property("photo_dir",lambdanode:len(node.children)==0)# Find the 3 largest photo directoriesphoto_dirs=[nodefornodeintreeifnode.property["photo_dir"]]photo_dirs.sort(key=node.property[PathTreeProperty.SIZE],reverse=True)print("The 3 largest photo directories:",photo_dirs[:3])# Export all the photo directories with the main information to csvtree.to_csv("large_photo_dirs",properties=["num_of_jpg",PathTreeProperty.SIZE],node_condition=lambdanode:node.property["photo_dir"])Quick analysis of the whole C volume in Windows to get the largest nodes,
exploiting the analytics module.# Generate the whole tree (could take a while if the volume is large)tree=PathTreeAnalytics("C:/")# Get nodes larger than 100 GB, if anyprint("Nodes larger than 100 GB")fornodeintree.get_large_nodes(limit=100*1024**3):print(node.path.as_posix())# Get the top 10 largest nodes (without ancestors)print("The top 10 largest nodes")fornodeintree.get_k_largest_nodes(k=10):print(f"{node.property[PathTreeProperty.SIMPLE_SIZE]}-{node.path.as_posix()}")