Package: astgen
astgen - A tool for generating Abstract Syntaxt Trees=====================================================Introduction------------This tools allows language and compiler developers to generate Abstract Syntax Trees that can beused by parsers (or parser generators) to record the result of a parse phase.   The input is a setof python classes which describe the methods and members of the AST classes specific to the parserand outputs language specific (so far Java and C++) classes with the definitions of the variousAST classes.Installation------------```pip install astgen```or from github```git clone [git-repo-url] astgencd astgenpython setup.py install```Getting Started---------------Upon installation, a script called "astgen" is created.  To see its usage, invoke it with the -h option:```bash-3.2$ astgen -hUsage: astgen [options]Options:-h, --help            show this help message and exit-o OUTPUTDIR, --outputdir=OUTPUTDIRThe output folder in which all generted files arewritten to.-l LAYOUT_BACKEND, --layout_backend=LAYOUT_BACKENDThe backend to decide the package layout of the codebeing generated, eg single file, multiple files etc-p PLATFORM_BACKEND, --platform_backend=PLATFORM_BACKENDThe backend to help with the platform to target codegenration for, eg java, python, stl, C etc-c BACKEND_CONFIG, --data=BACKEND_CONFIGData required any custom data that can be used/passedto the platform or layout backends.-m MODEL_FILE, --modelfile=MODEL_FILEInput model file containing AST definitions for whichAST code is to be generated.```The MODEL_FILE and BACKEND_CONFIG parameters are mandatory while the Platform parameter defaults to C++ (ie outputs are C++ classes) and a two file layout is used where all node declarations are writen to the header (.h) file and the definitions are written to the implementation file (.cpp).Sample Usage------------To try out the /alculator example:```cd samples/calculatorastgen -o /tmp/calculator -m model.py -c cpp_config.py```This would now generate all the files for the various AST nodesassociated with the sample in the folder /tmp/calculatorModels------Models define the structure of AST nodes that are to be generated.   A model looks similar to:```class NodeClass(ASTNode):properties = dict(member1 = MemberType1,member2 = MemberType2,...)```Nodes can also inherit from other defined nodes, eg:```Operator = EnumType("Operator", "PLUS", "MINUS", "MUL", "DIV")class Expression(ASTNode): passclass BinaryExpression(Expresssion):"""Has a binary operator and left and right sub expressions."""properties = dict(op = Operator,lhs = "Expression",rhs = "Expression")class UnaryExpression(Expression):"""Has a prefix operator and an expression"""properties = dict(op = Operator,child = "Expression")```Types will be discussed in next section.Config------The config file (specified with the -c parameter) contains overrides and other data used by the various parts of astgen while generating the AST code.  The different parameters depend on the layout and platform backends used and will be described in the respective sections.Types------In the Models section, nodes and properties were discussed.  Each property must have a type.  The type value of a property can be a String which indicates a reference to a value of another defined class (in the above case an "Expression") or it can be one of the following types:#### BasicType("type")Defines a native type that is platform specific and will resolved by the PlatformBackend object.  For instance:```Integer = BasicType("int")```defines a type called Integer which will is of type "int" and can be equal to an int variable in C/C++ or in Java.Note that this does not mean that only primitive types can be defined in this manner.  The use of BasicType is only a hint that the underlying type (and its string representation that is rendered in the final output) is determined by the PlatformBackend.  For a more complicated examle the following can also be defined:```PointListPtr = BasicType("PointListPtr")```and in return, this could be resolved to (in C++) as:```std::shared_ptr<std::list<Point>>```How these types are resolved will be discussed in the PlatformBackend section.#### EnumType(TypeName, EnumVal1, EnumVal2, EnumVal3....)Defines an Enum type similar to the following:```enum TypeName{EnumVal1,EnumVal2,EnumVal3...}```#### PairOf(Type1, Type2, TypeName = '')Defines a Pair type over the given two sub types.  In C++ this would be similar to:```typedef std::pair<Type1, Type2> TypeName```If the TypeName is not specified, a name is automatically generated and used.#### ListOf(BaseType, TypeName = '')Defines a type that is the List of a BaseType typed values.  In C++ this would be similar to:```typedef std::list<Type1, Type2> TypeName```If the TypeName is not specified, a name is automatically generated and used.#### MapOf(KeyType, ValueType)Defines a type that is the Map with the key and value types given by KeyType and ValueType respectively.  In C++ this would be similar to:```typedef std::map<KeyType, ValueType> TypeName```If the TypeName is not specified, a name is automatically generated and used.Platforms---------The Platform objects are responsible for handling and delegating all concerns related to the specific platform the ASTs are being generated for (eg C++, Java etc).   For now a Platform object only provides one method:```def evalType(self, typeobj): pass```This method is responsible for returning the string representation of a type object (eg those discussed in the Types section) specific to the platform.  To achieve the results above, the astgen.platforms.CPlusPlus backend looks like this:```class CPlusPlus(astgen.ASTPlatform):def evalType(self, typeobj):if type(typeobj) is str:return typeobj + "Ptr"elif type(typeobj) is astgen.BasicType:if typeobj.typename == "boolean": return "bool"if typeobj.typename == "string": return "std::string"return typeobj.typenameelif type(typeobj) is astgen.ListOf:return "std::list<%s>" % self.evalType(typeobj.base_type)elif type(typeobj) is astgen.PairOf:return "std::pair<%s,%s>" % (self.evalType(typeobj.type1), self.evalType(typeobj.type2))elif type(typeobj) is astgen.MapOf:return "std::map<%s,%s>" % (self.evalType(typeobj.key_type), self.evalType(typeobj.value_type))elif type(typeobj) is astgen.EnumType:return typeobj.enum_namereturn super(CPlusPlus, self).evalType(typeobj)```Custom platform backends can be provided to the astgen script with the -p parameter.Alternatively, it can be defined in the Config(.py) file itself.So far the following (hopefully self explanatory) platforms are defined:#### astgen.platforms.CPlusPlus#### astgen.platforms.JavaLayouts-------Regardless of the platform, there would be several ways to layout the generate nodes.  For instance, for C++ alone, the following (and several more) layouts are possible:- Monolithic header file: A single .h file that would contain all class definitions along with their implementations.- Two files: Broken down into one header file (containing all the interface/class declarations) and an implementation file (containing all the class/implementation definitions).- Two files per class: Each class with its own header and implementation files.There could be several other layouts depending on the needs of the project.  All these layouts inherit from the ASTLayout base.  The ASTLayout base has the following methods:```class ASTLayout(object):"""Given an AST node generates the code for the node.  This can be used togenerate AST code for different languages or platforms."""def orderNodes(self, nodes):"""Called to order all nodes in any way as deemed necessary."""passdef generationStarted(self, nodelist):"""Called before starting node generation for any of the nodes."""passdef generationFinished(self, nodelist):"""Called after the code generation of all nodes has completed."""passdef nodeStarted(self, node):"""Called before the generation of code for a particular node."""passdef renderNodes(self, nodelist):"""Called to render all nodes"""passdef renderNode(self, node):"""Called to render a particular node."""passdef nodeFinished(self, node):"""Called after the generation of code for a particular node."""pass```Shown below are the laout classes that are defined so far (along with the parameters they accept in the config file):##### astgen.layouts.OneFileLayoutThe monolithic one file layout where all class are declared and defined in the same file.  The following options in the config file are accepted:- HEADER_OUTPUT:    The output file to which all classes will be written to.- HEADER_TEMPLATE:  The (jinja2) template file that will be used to render all the classes (defaults to "cpp_header").##### astgen.layouts.TwoFilesLayoutThe two files layout where all class interface declarations are written to a header and implementations/definitions are written to an implementation file (eg .cpp, .m etc).  The following options in the config file are accepted:- HEADER_OUTPUT:            The output file to which all classes declarations will be written to.- HEADER_TEMPLATE:          The (jinja2) template file that will be used to render all the classes (defaults to "cpp_header").- IMPLEMENTATION_OUTPUT:    The output file to which all classes definitions/implementations will be written to.- IMPLEMENTATION_TEMPLATE:  The (jinja2) template file that will be used to render all the classes implementaitons/definitions (defaults to "cpp_implementation").##### astgen.layouts.TwoFilesPerNodeLayoutIn this layout, each node is written to its own file (typical<NodeName>.h and <NodeName.cpp>).  Additionally, a forward defs file, apublic headers file and a header file for all enums is also generated.These are denoted by:- FWDDEFS_OUTPUT:           The output file to which all forward definitions are written (optional).- ENUMS_OUTPUT:             The output file to which all enums are written (optional).- PUBLIC_OUTPUT:            The output file to which all include headers are written to (optional).- FWDDEFS_TEMPLATE:         The (jinja2) template file that will be used to render the forward definitions (if used).- ENUMS_TEMPLATE:           The (jinja2) template file that will be used to render the enum definitions (if used).- PUBLIC_TEMPLATE:          The (jinja2) template file that will be used to render the public header includes (if used).Note that the above are all optional and not required in all cases (forinstance for Java none of the above are required).