Package: saplings
saplingsis a static analysis tool for Python. Given a program,saplingswill build object hierarchies for every module imported in the program. Object hierarchies are dependency trees where the root node represents a module and each child represents an attribute of its parent. These can be useful fortype inferenceand mining data about how a module is used.Object hierarchies or attribute trees?Tracks modules by default, but build trees foranyobjectsWhy is it unique? Can track object flow through many complex paths through a program without executing itInstallationRequires Python 3.X.You can installsaplingswithpip:$pipinstallsaplingsUsageUsing saplings takes only two steps. First, convert your input program into anAbstract Syntax Tree (AST)using theastmodule. Then, import theSaplingsobject and initialize it with the root node of the AST.importastfromsaplingsimportSaplingsmy_program=open("path_to_your_program.py","r").read()program_ast=ast.parse(my_program)my_saplings=Saplings(program_ast)That's it. To access the object hierarchies, simply call theget_treesmethod in yourSaplingsobject, like so:my_saplings.get_trees()# => [ObjectNode(), ObjectNode(), ..., ObjectNode()]For more advanced usage of theSaplingsobject, read the docstringhere.Printing an Object Hierarchyget_treesreturns a list ofObjectNodes, each representing the root node of an object hierarchy and which has the following attributes:name(str):Name of the objectis_callable(bool):Whether the object is callable (i.e. has__call__defined)order(int):Indicates the type of connection to the parent node (e.g.0is an attribute of the parent,1is an attribute of the output of the parent when called, etc.);-1if node is rootfrequency(int):Number of times the object is referenced (i.e. appears) in the programchildren(list):List of child nodesTo pretty-print a tree, simply pass its root node into therender_treegenerator, like so:fromsaplingsimportrender_treetrees=my_saplings.get_trees()root_node=trees[0]forbranches,nodeinrender_tree(root_node):print(f"{branches}{node}")numpy (NC, -1)
 +-- random (NC, 0)
 |   +-- randn (C, 0)
 |       +-- __sub__ (C, 1)
 |       |   +-- shape (NC, 1)
 |       |   +-- __index__ (C, 1)
 |       +-- sum (C, 1)
 +-- matmul (C, 0)
 +-- expand_dims (C, 0)
     +-- T (NC, 1)(Here,NCmeans indicates a non-callable node andCa callable node.-1/0/1indicate the order of the node's connection to its parent).To create a dictionary representation of a tree, pass its root node into thedictify_treefunction, like so:fromsaplingsimportdictify_treedictify_tree(root_node){"numpy":{"is_callable":False,"order":-1,"frequency":1,"children":[{"random":...},{"matmul":...},{"expand_dims":...}]}}Interpreting the object hierarchyEach node is anobjectand an object can either becallable(i.e. has__call__defined) ornon-callable. Links between nodes each have anorder–– a number which describes the relationship between a node and its parent. If a node is a 0th-order child of its parent object, then it's an attribute of that object. If it's a 1st-order child, then it's an attribute of the output of the parent object when it's called, and so on. For example:my_parent=module.my_objmy_parent.attr# attr is a 0th-order child of my_objmy_parent().attr# attr is a 1st-order child of my_objmy_parent()().attr# attr is a 2nd-order child of my_objWhat counts as a function?In Python, subscripts, comparisons, and binary operations are all just syntactic sugar for function calls, and are treated by saplings as such. Here are some example "translations:"my_obj['my_sub']# => my_obj.__index__('my_sub')my_obj+10# => my_obj.__add__(10)my_obj==None# => my_obj.__eq__(None)LimitationsSaplingsstatically analyzesthe usage of a module in a program, meaning it doesn't actually execute any code. Instead, it traverses the program's AST and tracks "object flow," i.e. how an object is passed through a program via variable assignments and calls of user-defined functions and classes. To demonstrate this idea, consider this example ofcurryingand the tree saplings produces:importtorchdefcompose(g,f):defh(x):returng(f(x))returnhdefF(x):returnx.TdefG(x):returnx.sum()composed_func=compose(F,G)composed_func(torch.tensor())Saplings identifiestensoras an attribute oftorch, then follows the object as it's passed intocomposed_func. Because saplings has an understanding of howcomposed_funcis defined, it can analyze how the object is usedwithinthe function and capture theTandsumsub-attributes.While saplings can track object flow through many complex paths in a program, I haven't tested every edge case, and there are some situations where saplings produces inaccurate trees. Below is a list of all the failure modes I'm aware of (and currently working on fixing). If you discover a bug or missing feature that isn't listed here, pleasecreate an issuefor it.Data StructuresAs of right now, saplings can't trackassignmentsto comprehensions, generator expressions, dictionaries, lists, tuples, or sets. It can, however, track object flowinsidethese data structures. For example, consider the following:importnumpyasnpvectors=[np.array([0]),np.array([1]),np.array([2])]vectors[0].mean()Saplings can capturearrayand add it to thenumpyobject hierarchy, but it cannot capturemean, and thus produces the following tree:This limitation can have some unexpected consequences. For example, functions that return multiple values with onereturnstatement (e.g.return a, b, c) are actually returning tuples. Therefore, the output of those functions won't be tracked by saplings. The same logic applies to variable unpacking with*and**.Control FlowHandling control flow is tricky. Tracking object flow in loops and conditionals requires making assumptions about what code actually executes. For example, consider the following:importnumpyasnpforxinnp.array([]):print(x.mean())Because saplings only doesstaticanalysis and not type inference, it doesn't know thatnp.array([])is an empty list, and that therefore the loop never executes. In this situation, capturingmeanand adding the__index__ -> meansubtree tonumpy -> arraywould be a false positive, sincex(i.e. the output ofnp.array().__iter__()) is never defined. To handle this, saplingsshouldbranch out and produce two possible trees for this module –– one that assumes the loop doesn't execute, and one that assumes it does:But as of right now, saplings will only produce the tree on the right –– that is, we assume the bodies offorloops are always executed (because they usually are).Below are the assumptions saplings makes for other control flow elements.whileloopswhileloops are processed under the same assumption asforloops –– that is, the body of the loop is assumed to execute, even if the loop condition evaluates toFalse.if/elseblocksSaplings processesifandelseblocks more conservatively than loops. It tracks object flow within these blocks but doesn't allow changes to the namespace to persist into the parent scope. For example, given:importnumpyasnpX=np.array()ifcondition:X=np.matrix()else:X.mean()X=NoneY=np.array()print(X.sum())print(Y.max())saplings will produce the following tree:Notice how the value ofXbecomes ambiguous after we exit theifblock, since we don't know ifconditionevaluates toTrueorFalse. To handle this, saplings simply stops tracking any variable that's defined in the outer scope, likeX, if it's modified inside anif/elseblock. Similarly, notice how there exists an execution path whereYis never defined andY.max()throws an error. To handle this, saplings assumes that any variable defined inside anif/elseblock, such asY, doesn't persist into the outer scope.Both of these assumptions are made in attempt to reduce false positives and false negatives. But ideally, saplings would branch out and produce two separate trees for this module –– one that assumes theifblock executes and another that assumes theelseblock executes, like so:try/exceptblockstry/exceptblocks are handled similarly toif/elseblocks –– that is, changes to the namespace made in either block do not persist in the outer scope.Notably,tryandelseblocks are treated as a single block, sinceelseis only executed iftryexecutes without exceptions. Andfinallyblocks are treated as separate from the control flow, since code in here always executes regardless of whether an exception is thrown.return,break, andcontinuestatementsAll code underneath areturn,break, orcontinuestatement is assumed not to execute and will not be analyzed. For example, consider this:importnumpyasnpforxinrange(10):y=np.array([x])continuey.mean()It may be the case thatmeanis actually an attribute ofnp.array, but saplings will not capture this sincey.mean()is never executed.Notably, saplings doesn't apply this assumption to statements inside control flow blocks. For example, if thecontinuestatement above was changed to:ifcondition:continueThenmeanwouldbe captured by saplings as an attribute ofnp.array.FunctionsRecursionSaplings cannot process recursive function calls. Consider the following example:importsome_moduledefmy_recursive_func(input):ifinput>5:returnmy_recursive_func(input-1)elifinput>1:returnsome_module.fooelse:returnsome_module.baroutput=my_recursive_func(5)output.attr()We know this function returnssome_module.foo, but saplings cannot tell which base case is hit, and therefore can't track the output. To avoid false positives, we assume this function returns nothing, and thusattrwill not be captured and added to the object hierarchy. The tree saplings produces is:GeneratorsGenerators aren't processed as iterables. Instead, saplings ignoresyield/yield fromstatements and treats the generator like a normal function. For example, given:importsome_moduledefmy_generator():yield fromsome_module.some_itemsforiteminmy_generator():print(item.name)__index__ -> namewon't be added as a subtree tosome_module -> some_items, and so the tree produced by saplings will look like this:Notably, this limitation will only produce false negatives –– not false positives.Anonymous FunctionsWhile thebodiesof anonymous (lambda) functions are processed, object flow through assignments and calls of those functions is not tracked. For example, given:importnumpyasnptrans_diag=lambdax:np.diagonal(x.T)trans_diag(np.random.randn(5,5))saplings will produce the following tree:Notice thatTis not captured as an attribute ofnumpy.random.randn, butdiagonalis captured as an attribute ofnumpy. This is because the body of thelambdafunction is processed by saplings, but the assignment totrans_diagis not recognized, and therefore the function call is not processed.ClassesSaplings can track object flow in static, class, and instance methods, getter and setter methods, class and instance variables, classes defined within classes, and class closures (i.e. functions that return classes). Notably, it can keep track of the state of each instance of a user-defined class. Consider the following program and the tree saplings produces:importtorch.nnasnnfromtorchimporttensorclassPerceptron(nn.Module):loss=Nonedef__init__(self,in_channels,out_channels):super(Perceptron,self).__init__()self.layer=nn.Linear(in_channels,out_channels)self.output=Perceptron.create_output_layer()@staticmethoddefcreate_output_layer():deflayer(x):returnx.mean()returnlayer@classmethoddefcalculate_loss(cls,output,target):cls.loss=output-targetreturncls.lossdef__call__(self,x):x=self.layer(x)returnself.output(x)model=Perceptron(1,8)output=model(tensor([10]))loss=Perceptron.calculate_loss(output,8)While saplings can handle many common usage patterns for user-defined classes, such as the ones above, there are some things saplings can't handle yet. Below are all the limitations I'm aware of:Class ModificationsIn the example above, calling the class methodPerceptron.calculate_lossshould change the value of the class variableloss. However, saplings cannot track modifications to a class when it's passed into a function. Saplingscanhandle when a class is modified in the scope in which it was defined, like so:Perceptron.loss=tensor()Perceptron.loss.item()Here,itemwould be captured and added to the tree as an attribute oftensor. But if the class is modified via an alias, like so:NeuralNet=PerceptronNeuralNet.loss=tensor()Perceptron.loss.item()Then saplings won't captureitem. Saplings also can't propagate class modifications to existing instances of the class. For example, continuing the code above:model=Perceptron(1,8)Perceptron.loss=tensor()model.loss.item()Because the change toloss, a class variable, won't propagate tomodel, an instance ofPerceptron,itemwon't be captured as an attribute oftensor.InheritanceSaplings cannot recognize inherited methods or variables in user-defined classes. For example, given:importsome_moduleclassMyClass(module.Foo):def__init__(self,x):self.bar(x)saplings will not recognizebaras an attribute ofmodule.Foo, despitebarbeing an inherited method. This limitation also holds true when the base class is user-defined.MetaclassesOnce I learn what metaclasses actually are and how to use them, I'll get around to handling them in saplings. But for now this is on the bottom of my to-do list since 99.9% of Python users also don't know what the hell metaclasses are.Miscellaneousglobalandnonlocalstatementsglobalstatement are used inside functions to declare a variable to be in the global namespace. But saplings doesn't recognize these statements and change the namespace accordingly. For example, given:importsome_modulemy_var=some_module.foodefmy_func():globalmy_varmy_var=Nonemy_func()my_var.bar()saplings will produce a tree withbaras an attribute offoo. This would be a false positive since callingmy_funcsetsmy_vartoNone, and of courseNonedoesn't havebaras an attribute.nonlocalstatements are similar toglobals, except they allow you to modify variables declared in outer scopes. And likeglobals, saplings doesn't recognizenonlocalstatements.Built-in functionsNone of Python'sbuilt-in functionsare recognized by saplings. For example, consider theenumeratefunction:importsome_moduleforindex,iteminenumerate(some_module.items):print(item.some_attr)saplings won't captureattras an attribute ofsome_module.items.__iter__, which it would have ifsome_module.itemswasn't wrapped byenumerate.