# ROSE Compiler Framework/OpenMP Support
## Documentation

Official documentations
* The ROSE manual has a chapter (Chapter 12 OpenMP Support) explaining the details. pdf
* A paper was published for the uniqueness of the ROSE OpenMP Implementation pdf
* A paper about the Accelerator (GPU) support: "Early experiences with the openmp accelerator model."

## OpenMP Features

ROSE supports OpenMP 3.0 for C/C++ (and limited Fortran support) and has an experimental OpenMP Acclerator Model Implementation from OpenMP 4.0
* OpenMP Acclerator Model Implementation
Connection point to the frontend is located in sage_support.cpp, via OmpSupport::processOpenMP called by SgFile::secondaryPassOverSourceFile
void processOpenMP(SgSourceFile *sageFilePtr) of ompAstConstruction.cpp // top level driver of OpenMP processing in ROSE
* Frontend parsing source files (ompparser.yy and ompFortranParser.C) are located under https://github.com/rose-compiler/rose/tree/master/src/frontend/SageIII
** void build_OpenMP_AST(SgSourceFile *sageFilePtr) of ompAstConstruction.cpp // process pragmas and build AST for OpenMP
*** void attachOmpAttributeInfo(SgSourceFile *sageFilePtr) of ompAstConstruction.cpp // find and parse OpenMP pragmas into OmpAttribute info.
*** convert_OpenMP_pragma_to_AST // convert attributes into dedicated OpenMP AST nodes
* The transformation of OpenMP into threaded code is located in omp_lowering.cpp, under https://github.com/rose-compiler/rose/blob/master/src/midend/programTransformation/ompLowering
** lower_omp
* The OpenMP runtime interface is defined in libxomp.h and xomp.c under the same ompLowering directory mentioned above
** GCC's OpenMP runtime library, libgomp is used with this implementation.
** Optionally, other runtime libraries, such as Omni runtime library can be hooked with ROSE. But this support is not being actively maintained.
The OpenMP processing happens in void SgFile::secondaryPassOverSourceFile of rose/src/frontend/SageIII/sage_support/sage_support.cpp. This pass collects extra information about the source file such as comments, preprocessing directives, etc.

## Configuration

Configuration: please always try to use --with-gomp_omp_runtime_library=/usr/apps/gcc/4.4.1/lib64/ when configuring ROSE. So the generated ROSE translators can automatically link with libgomp.a to generate executables for you. This will also allow the execution tests of the omp Lowering be executed to catch errors. Without this option, only the compile level tests will run.

## Command line options

Like all other compilers supporting OpenMP, you have to explicitly turn on this support by using -rose:openmp:lowering, for example:
This option tells a ROSE translator to recognize OpenMP pragmas and translate the input to use libgomp runtime library. Detailed command lines for compilation, backend compiler , and linking can be seen by using a verbose flag
Three internal steps will show up in this verbose mode:
* ROSE translator's translation: identityTranslator -rose:openmp:lowering -rose:verbose 3 ...-DUSE_ROSE --c -D_OPENMP your_openmp_code.c
* Backend compiler compilation: gcc -DUSE_ROSE -D_OPENMP -I/export/tmp.liao6/workspace/masterClean/build64/install/include rose_your_openmp_code.c -c -o mg_your_openmp_code.o
* Linking: gcc your_openmp_code.o -lm /export/tmp.liao6/workspace/masterClean/build64/install/lib/libxomp.a /usr/apps/gcc/4.4.1/lib64//libgomp.a -lpthread
More command line option choices for OpenMP support
* -rose:OpenMP, -rose:openmp follow OpenMP 3.0 specification for C/C++ and Fortran, perform one of the following actions:
* -rose:OpenMP:parse_only, -rose:openmp:parse_only parse OpenMP directives to OmpAttributes, no further actions (default behavior now)
* -rose:OpenMP:ast_only, -rose:openmp:ast_only on top of -rose:openmp:parse_only, build OpenMP AST nodes from OmpAttributes, no further actions
* -rose:OpenMP:lowering, -rose:openmp:lowering on top of -rose:openmp:ast_only, transform AST with OpenMP nodes into multithreaded code targeting GCC GOMP runtime library

### internal steps

compilation
* ./identityTranslator -I../../../../../../rose-develop/src/frontend/SageIII -I../../../../../../rose-develop/src/midend/programTransformation/ompLowering -I../../../../.. -rose:openmp:lowering -g --edg:no_warnings -c ../../../../../../rose-develop/tests/nonsmoke/functional/CompileTests/OpenMP_tests/pi.c
Linking:

## Testing

Testing
* There are about 70 builtin execution tests (many have self-verification) in ROSE.
** OpenMP parsing tests: tests/CompileTests/OpenMP_tests
** OpenMP translation and execution tests: rose/tests/roseTests/ompLoweringTests, need --with-gomp_omp_runtime_library=/usr/apps/gcc/4.4.x(or later)/lib64/ to link and run the tests.
Some benchmarks are used to test OpenMP support in ROSE in Jenkins (our regression test server)
* a22b-NPB-2.3-C-parallel: all 8 benchmarks pass
* a21-SPEC-OMP-64bit-parallel: 3 benchmarks pass.
* LULESH OpenMP version: download
For builtin test:
* source files are in: https://github.com/rose-compiler/rose/tree/master/tests/nonsmoke/functional/CompileTests/OpenMP_tests
* the test driver is defined in https://github.com/rose-compiler/rose/tree/master/tests/nonsmoke/functional/roseTests/ompLoweringTests/Makefile.am . See the list of files under C_TESTCODES_REQUIRED_TO_RUN .
You have to configure the path to GOMP if you want to them to be automatically executed when "make check" is typed. e.g. ../sourcetree ... --with-gomp_omp_runtime_library=/usr/apps/gcc/4.4.1/lib64/

## Parsing OpenMP directives

Frontend parsing source files (ompparser.yy and ompFortranParser.C) are located under https://github.com/rose-compiler/rose/tree/master/src/frontend/SageIII
* C/C++ directives are parsed using Flex/Bison:
** Lexer https://github.com/rose-compiler/rose/blob/master/src/frontend/SageIII/omplexer.ll
** https://github.com/rose-compiler/rose/blob/master/src/frontend/SageIII/ompparser.yy
* Fortran directives are parsed by a hand-written recursive descent parser https://github.com/rose-compiler/rose/blob/master/src/frontend/SageIII/ompFortranParser.C
* The driver to call the parsers: void processingOpenMP(SgSoureFile*) from https://github.com/rose-compiler/rose/blob/master/src/frontend/SageIII/ompAstConstruction.cpp
** void attachOmpAttributeInfo(SgSourceFile *sageFilePtr) // call parse_fortran_openmp
To recognize a new keyword, for example, a clause : target
* add a token name in ompparser.yy : %token TARGET
* parse the token in omplexer.ll: target {return cond_return ( TARGET ); }
* add grammar rules into ompparser.yy

### OmpAttribute

The parsing results are stored into persistent AST attribute. A derived type called OmpAttribute is used.
./src/frontend/SageIII/OmpAttribute.h /.C

## AST support for OpenMP

We define a set of dedicated AST nodes to represent OpenMP directives in your source code. For an example input code, you can view the AST using dotGeneratorWholeASTGraph
./dotGeneratorWholeASTGraph -c -rose:OpenMP:ast_only reduction_1.c
The generated dot graph will have the following nodes
* SgBasicBlock // the basic block of the function definition
** SgOmpParallelStatement // dedicated OpenMP parallel statement in AST
*** SgOmpForStatement // dedicated Omp for statement. It has SgOmpReductionClause and SgOmpPrivateClause, obtained from ->get_clauses
**** SgForStatement // the actual loop affected by "omp parallel for"
The combined directives like "omp parallel for " is split into two directives "omp parallel" and "omp for". All clauses are attached to the inner level directive as much as possible. This is to support simpler handling later on (like in OpenMP lowering).
* Note: there is no dedicated AST nodes for combined directives for now. But we can add them if necessary.

### Omp Nodes

A list of the OpenMP AST nodes are (from http://rosecompiler.org/ROSE_HTML_Reference/annotated.html)
* SgOmpAlignedClause
* SgOmpAtomicStatement
* SgOmpBarrierStatement
* SgOmpBeginClause
* SgOmpBodyStatement
* SgOmpClause
* SgOmpClauseBodyStatement
* SgOmpCollapseClause
* SgOmpCopyinClause
* SgOmpCopyprivateClause
* SgOmpCriticalStatement
* SgOmpDefaultClause
* SgOmpDeviceClause
* SgOmpDoStatement
* SgOmpEndClause
* SgOmpExpressionClause
* SgOmpFirstprivateClause
* SgOmpFlushStatement
* SgOmpForStatement
* SgOmpIfClause
* SgOmpLastprivateClause
* SgOmpLinearClause
* SgOmpMapClause
* SgOmpMasterStatement
* SgOmpNowaitClause
* SgOmpNumThreadsClause
* SgOmpOrderedClause
* SgOmpOrderedStatement
* SgOmpParallelStatement
* SgOmpPrivateClause
* SgOmpReductionClause
* SgOmpSafelenClause
* SgOmpScheduleClause
* SgOmpSectionsStatement
* SgOmpSectionStatement
* SgOmpSharedClause
* SgOmpSimdStatement
* SgOmpSingleStatement
* SgOmpTargetDataStatement
* SgOmpTargetStatement
* SgOmpTaskStatement
* SgOmpTaskwaitStatement
* SgOmpThreadprivateStatement
* SgOmpUniformClause
* SgOmpUntiedClause
* SgOmpVariablesClause
* SgOmpWorkshareStatement
Please refer to the Doxygen documentation for the member access functions.

### AST Graph

To view the AST graphs of your input OpenMP codes, you should use either dotGraphGeneratorWhole of pdfGenerator .
One example dot graph is available at parallefor.dot.png within https://github.com/chunhualiao/rose-ast .

### Access OpenMP clauses and variables

Please refer to the Doxygen documentation for the member access functions of SgOmp* nodes.
Many helper functions are defined in the OmpSupport namespace, as shown in http://rosecompiler.org/ROSE_HTML_Reference/namespaceOmpSupport.html
* https://github.com/rose-compiler/rose/blob/master/src/frontend/SageIII/OmpAttribute.h
* https://github.com/rose-compiler/rose/blob/master/src/midend/programTransformation/ompLowering/omp_lowering.h
The best examples of how to access those nodes' information are in the OpenMP lowering step of ROSE.
* https://github.com/rose-compiler/rose/blob/master/src/midend/programTransformation/ompLowering/omp_lowering.cpp
For example, to obtain the list of variable references in threadprivate clause, the code looks like
Another example to check if a variable is within some OpenMP clauses' variable list

### Query data sharing attributes

There is an interface function to query data sharing attribute of a variable referenced in an OpenMP program, declared in https://github.com/rose-compiler/rose/blob/master/src/midend/programTransformation/ompLowering/omp_lowering.h
Test/Example code
* https://github.com/rose-compiler/rose/blob/master/tests/nonsmoke/unit/getDataSharingAttribute.C
Testing command line:

### unparsing of AST nodes

They are defined in
src/backend/unparser/languageIndependenceSupport/unparseLanguageIndependentConstructs.h
src/backend/unparser/languageIndependenceSupport/unparseLanguageIndependentConstructs.C

## Enabling OpenMP processing from your translator
### Enabling OpenMP parsing and AST creation

The simplest and recommended way is to pass command line option "-rose:OpenMP:ast_only" to your translator.
* This will turn on parsing OpenMP directives and creating dedicated OpenMP AST nodes. Users have the choice of turning on/off of OpenMP parsing.
The 2nd option (recommended) is that you can push the option internally in your translator
The third option (not really recommended), you can replicate all the internally handling in your translator, as shown in https://github.com/rose-compiler/rose/blob/master/src/frontend/SageIII/sage_support/cmdline.cpp and https://github.com/rose-compiler/rose/blob/master/src/frontend/SageIII/sage_support/sage_support.cpp
* this is a bit complex since you are now exposed to both Project and File level internal flags.
* you have to understand details of how several flags work together to control parsing, ast creation, and lowering.
* and explicitly call the OpenMP processing function
* and pass "-DOPENMP" to backend compiler if backend compiler is needed.

### Enabling OpenMP Lowering

Sometimes it is desired to call OpenMP lowering from your own translator, not from the builtin frontend of ROSE. Here is an example:

## Instrumentation support

XOMP (ROSE's OpenMP runtime layer) supports a simple event instrumentation to collect timestamps of key OpenMP events. The data collected can be helpful for further analysis, such as investigating serial vs. parallel regions during an execution.
Environment variable to turn on this support
* XOMP_REGION_INSTR =0|1: simply expose this environment variable will turn on this support. This option is turned off by default to avoid unnecessary overhead.
** in BASH: type "export XOMP_REGION_INSTR=1"
Once this feature is turned on, use rose to compile your OpenMP code. The executable will generate a data file with a time stamp as its prefix, such as "2013_03_05_10_05_45.jacobi". This file will contain a time stamp, value 1 or 2, and file location for each change between serial and parallel execution.
Sample GNU plot script to generate a figure from the data file generated by XOMP
Sample generated GNU plot figure: X-axis is the timeline, Y-axis is for the serial (value 1) vs parallel (value 2) execution
Relevant commits for this support
* https://github.com/rose-compiler/edg4x-rose/commit/59233fc54208fbb8ad78e24eb76266cec08ee55e , latest
* https://github.com/rose-compiler/edg4x-rose/commit/3f0a87daa6ac24b4cd22fadbef5b4a507317c6bf,
* https://github.com/rose-compiler/edg4x-rose/commit/ff1ab8601365d400f730ea5606f3b4f44316c72a, earliest commit

## How large is the implementation?

Data collected on July 7th, 2015
Directly source lines
Indirect support source files
The OpenMP implementation in ROSE supports OpenMP 3.0 and the accelerator (GPU) support in OpenMP 4.0. It has three components: a frontend, a middle end, and a runtime layer, with 4849, 6151, and 6203 lines of code respectively. In total, there are 17203 lines of code which are dedicated for implementing OpenMP.
In addition, two other supportive components in ROSE are used to help OpenMP implementation: 1) the AST outliner (extracting portions of code into functions) and 2) the AST interface functions (building and manipulating AST). The AST outliner has 17068 lines of code. The AST interface functions have 42262 lines of code. So the lines of indirect source files are 42262.
Counting both direct and indirect source files, there are 76533 lines related to OpenMP implementation in ROSE.
Using an online software construction cost calculator (http://www.csgnetwork.com/costconstrmodelcalc.html) assuming a developer can write and maintain 500 lines of code per month. we estimate that the cost is 48 man-months (or 480K using a 10K monthly cost rate per person) for developing the dedicated OpenMP source files of 17.2K source lines. The total cost for both direct and indirect source files is estimated to be 228 man-months or 2.28 millions.
We have not tried to add the parsing and unparsing components in ROSE into this estimation. The roughly estimation is that there are around 1 million lines of code to parse input code and later unparse the source code as output.

## Publications

* Chunhua Liao, Yonghong Yan, Bronis R. de Supinski, Daniel J. Quinlan, and Barbara Chapman. "Early experiences with the openmp accelerator model." In OpenMP in the Era of Low Power Devices and Accelerators, pp. 84-98. Springer Berlin Heidelberg, 2013.
* Chunhua Liao, Daniel J. Quinlan , Thomas Panas and Bronis de Supinski, A ROSE-based OpenMP 3.0 Research Compiler Supporting Multiple Runtime Libraries, the 6th International Workshop on OpenMP (IWOMP), June 14-16, 2010, Tsukuba, Japan. LLNL-CONF-422873