Package: open-fortran-parser
Extension of Java-based Open Fortran Parser and a Python wrapper enabling Fortran parsing from Python.Implementation has 2 parts: the XML generator written in Java, and Python wrapper for the generator.The implementation is tested on Linux, OS X and Windows.In this file, first the AST specification is described, then the Java implementation,
and then the Python wrapper.ContentsAST specificationComments and directivesProgramDeclarationsImplicit declarationVariable declarationExternal declarationSave declarationUseCompound statementsIfLoopSelectSimple statementsAssignmentCallExpressionsUnary operationMultiary operationSubroutineFunctionModuleWork in progressUnhandled corner casesJava XML generator for OFPdependencieshow to buildhow to runPython wrapper for the generatordependencieshow to buildhow to installhow to runas scriptas librarytestingcode coverageAST specificationFor any Fortran file, the resulting XML file has the following structure:<?xml version="1.0" encoding="UTF-8" standalone="no"?><ofpversion="0.8.5-1"><!-- version of the Open Fortran Parser used --><filepath="/path/to/parsed/file.f90"><!-- Fortran syntax goes here --></file></ofp>Root node is<ofp>, it has one subnode<file>.Inside the<file>, there might be one or many of the following nodes:<program><subroutine><function><module><interface>…Additionally, every XML node that was built using tokens from the source code
(which means almost any XML node) has its source code location described in the following way:<nodecol_begin="..."col_end="..."line_begin="..."line_end="..."/>For simplicity, the above XML file boilerplate as well as locations are stripped
from the examples that follow.For each presented construct, Fortran code snippet and corresponding XML AST is given.Comments and directivesComment:! my comment
!$omp parallel do<commenttext="! my comment"/><commenttext="!$omp parallel do"/>Directive:#define NDIMS 3<directivetext="#define NDIMS 3"/>Nodes<comment>and<directive>exist to carry comments and preprocessor directives, respectively.
These nodes might be in principle inserted before, after or within any of other nodes,
however, in practice they are either surrounding the top-level nodes (e.g. program or subroutine)
or are placed in-between non-compound declarations and/or statements within them.Note: compiler directives are comments in Fortran.Programprogramempty...end programempty<programname="empty"><body>...</body></program>In the body,Declarationsfollowed by any number of statements can be found.And each of the statements listed after the declarations,
can be one ofSimple statementsorCompound statements.DeclarationsA special node<specification>wraps all declarations:<specificationdeclarations="0"implicits="0"imports="0"uses="0">...</specification>It provides counts for each of the declaration type and contains a collection of declarations,
which can any of the following:<use><declaraion>…The<declaraion>node is special in a sense that it has type attribute that specifies
what kind of declaration it is.Implicit declarationimplicit none
implicitreal(A-H,O-Z)<declarationsubtype="none"type="implicit"/><declarationsubtype="some"type="implicit"><typename="real"type="intrinsic"/><letter-ranges><letter-rangebegin="A"end="H"/><letter-rangebegin="O"end="Z"/></letter-ranges></declaration>Variable declarationintegeri,j<declarationtype="variable"><typename="integer"type="intrinsic"/><variablescount="2"><variablename="i"/><variablename="j"/></variables></declaration>External declarationexternalomp_get_num_procsSave declarationsavenUseusempiusemy_interface,only:subroutine1,subroutine2use,non_intrinsic::my_moduleuse,intrinsic::iso_c_binding,only:c_int,c_float<usename="mpi"/><usename="my_interface"><only><nameid="subroutine1"/><nameid="subroutine2"/></only></use><usename="my_module"><naturename="non_intrinsic"/></use><usename="iso_c_binding"><naturename="intrinsic"/><only><nameid="c_int"/><nameid="c_float"/></only></use>Compound statementsCompound statements, e.g.:<if><loop><select>…each have<header>and<body>.IfIn the header of<if>, an expression is present.SeeExpressionsfor a definition.LoopIn the header of the<loop>, at least one<index-variable>is present.
It has<lower-bound>,<upper-bound>and<step>.SelectIn the body of<select>there multiple<case>nodes.
These are also compound (i.e. each of them has<header>and<body>),
however they exist only within the body of select statement.Simple statements<statement>...</statement>All simple statements are using<statement>node, which wraps around nodes like:<assignment><pointer-assignment><call><open><close><write><format><print><allocate><deallocate><return><stop><continue><cycle><arithmetic-if>…Assignmentx=1<assignment><target><nameid="i"/></target><value><literaltype="int"value="1"/></value></assignment>Callcallconfigurecallinitialize()callcalculate(1,2)callsomething(thing=my_value)<call><namehasSubscripts="false"id="configure"type="procedure"/></call><call><namehasSubscripts="true"id="initialize"type="procedure"><subscriptscount="0"/></name></call><call><namehasSubscripts="true"id="calculate"type="procedure"><subscriptscount="2"><subscripttype="simple"><literaltype="int"value="1"/></subscript><subscripttype="simple"><literaltype="int"value="2"/></subscript></subscripts></name></call><call><namehasSubscripts="true"id="something"type="procedure"><subscriptscount="1"><argumentname="thing"><nameid="my_value"/></argument></subscripts></name></call>ExpressionsExpression might be a single node like:<name><literal>…More complex expressions are built from the<operation>nodes, each of which contains
a collection of<operand>and<operator>nodes. Each operand contains an expression.Unary operation.not.flag<operationtype="unary"><operatoroperator=".not."/><operand><nameid="flag"/></operand></operation>Multiary operation'Hello'//' world'5+x<operationtype="multiary"><operand><literaltype="char"value="'Hello'"/></operand><operatoroperator="//"/><operand><literaltype="char"value="' world'"/></operand></operation><operationtype="multiary"><operand><literaltype="int"value="5"/></operand><operatoroperator="+"/><operand><nameid="x"/></operand></operation>SubroutineMany complex nodes contain<header>and<body>.The contents of the header depend on the type of the node. For example, in case of subroutines,
it contains list of parameters.Functionfunctionfoo...end functionfoo<functionname="foo"><header>...</header><body>...</body></function>Modulemoduleabcintegeri...contains
  subroutinesub()...end subroutinesub...end moduleabc<modulename="abc"><body><specificationdeclarations="1"implicits="0"imports="0"uses="0"><declarationtype="variable"><typename="integer"type="intrinsic"/><variablescount="1"><variablename="i"/></variables></declaration></specification>...</body><members><subroutinename="sub"><header/><body>...</body></subroutine>...</members></module>Work in progressRemaining details of AST are not decided yet. For the time being, to see implementation details,
please take a look intosrc/fortran/ofp/XMLPrinter.java.Unhandled corner casesin certain corner cases, the parse tree might deviate from the above description.This might be due to two main reasons:Some feature is not yet implemented in this XML output generatorThe events provided by OFP are not sufficient to generate a correct tree.In case 1, all contributions to this project are very welcome. The implementation of any one
of the missing features might not be very troublesome. The main reason why many of those features
are not implemented yet is because the Fortran codes the current contributors work with
do not use them.In case 2, there is a need to dynamically reorder/modify/delete nodes, or otherwise manipulate
existing parse tree while adding new nodes. Contributions are also very welcome,
but implementation might be much more challenging in this case.Java XML generator for OFPThis is an extension of Open Fortran Parser (OFP), which outputs abstract syntaxt tree (AST)
of parsed Fortran file in XML format - to a file or toSystem.out.dependenciesJava 1.7 or laterOpen Fortran Parser 0.8.5-1https://github.com/mbdevpl/open-fortran-parser/releasesThis is a patched version of OFP. The list of changes is available at the above link.ANTRL 3.5.2 (dependency of Open Fortran Parser)http://www.antlr3.org/download/Apache Commons CLI 1.4 or laterhttps://commons.apache.org/proper/commons-cli/download_cli.cgihow to buildGet dependencies, either manually, or using the provided script:pip3install-U-rrequirements.txtpython3-mopen_fortran_parser--depsexportCLASSPATH="${CLASSPATH}:$(pwd)/lib/*"Build:antexportCLASSPATH="${CLASSPATH}:$(pwd)/dist/*"This will create a.jarfile indistdirectory, and add it to the Java classpath.If you use a different python executable to install requirements, please provide it to ant too:ant-Dpython=/custom/pythonBecause the build script by default relies on “python3” executable.how to runjavafortran.ofp.FrontEnd--classfortran.ofp.XMLPrinter\--outputoutput.xml--verbosity0~100input.fwhere:The--verbosityflag controls verbosity of the parse tree. Defaluts to100when omitted.Maximum,100, means that all details picked up by Open Fortran Parser
will be preserved.Minimum,0, means that tree will contain only what is needed to reconstruct
the program without changing it’s meaning.The--outputflag controls where the XML should be written. Defaults to standard output
when omitted.and remaining command-line options are exactly as defined in OFP 0.8.5.To parsesome_fortran_file.fand save XML output intree.xmlwith minimum verbosity:javafortran.ofp.FrontEnd--classfortran.ofp.XMLPrinter\--outputtree.xml--verbosity0some_fortran_file.fAnd to dump XML with maximum verbosity to console:javafortran.ofp.FrontEnd--classfortran.ofp.XMLPrinter\--verbosity100some_fortran_file.fPython wrapper for the generatorUsing the wrapper should not require any special knowledge about the generator itself, other than
knowing the abstract syntax tree (AST) specification.dependenciesJava XML generator for OFP and all of its dependencies.Python version 3.5 or later.Python libraries as specified inrequirements.txt.Building and running tests additionally requires packages listed intest_requirements.txt.how to buildpip3install-U-rtest_requirements.txtpython3setup.pysdist--formats=gztar,zippython3setup.pybdist_wheelhow to installYou can simply install from PyPI:pip3installopen-fortran-parserOr using any of below commands, when installing from source:pip3install.pip3installdist/<filename>.whlpip3installdist/<filename>.tar.gzpip3installdist/<filename>.ziphow to runThe wrapper can be used as a script, or as a library.When running any installed version, even if installed from source, dependencies are automatically
installed together with the wrapper.Before running from source (without installation), however, please follow “how to build” section
for Java implementation above.
You can make sure that dependencies are configured correctly by running:python3-mopen_fortran_parser--check-depsIf the depenencies changed since you first ran the wrapper from the source tree, you can cleanup
outdated dependencies by executing:python3-mopen_fortran_parser--cleanup-depsas script$ python3 -m open_fortran_parser -h
usage: open_fortran_parser [-h] [--version] [-v VERBOSITY]
                           [--check-dependencies]
                           [input] [output]

Python wrapper around XML generator for Open Fortran Parser

positional arguments:
  input                 path to Fortran source code file (default: None)
  output                writable path for where to store resulting XML,
                        defaults to stdout if no path provided (default: None)

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  -v VERBOSITY, --verbosity VERBOSITY
                        level of verbosity, from 0 to 100 (default: 100)
  --check-dependencies, --check-deps
                        check if all required dependencies are present and
                        exit (default: False)

Copyright 2017-2019 by the contributors, Apache License 2.0,
https://github.com/mbdevpl/open-fortran-parser-xmlas libraryfromopen_fortran_parserimportparsexml=parse('my_legacy_code.f',verbosity=0)More examples available inexamples.ipynb.testingRun basic tests:python3-munittest-vTEST_LONG=1python3-munittest-v# this might take a long time...code coverageGetting code coverage results for Java requires JaCoCo agent, and JaCoCo CLI,
and both are dowonloaded automatically along with other development dependencies.Currently, test setup relies on JaCoCo 0.8.3:JaCoCo agent 0.8.3 (runtime)JaCoCo CLI 0.8.3 (nodeps)Run all test and gather code coverage:TEST_LONG=1TEST_COVERAGE=1python3-mcoveragerun--branch--source.-munittest-vThis will take a long while.Then, generate results for Python code:python3-mcoveragereport--show-missingpython3-mcoveragehtmlFinally, generate results for Java code:java-jar"lib/org.jacoco.cli-0.8.3-nodeps.jar"report"jacoco.exec"--classfiles"bin/"--sourcefiles"src/"--htmljacoco