etmc/tmLQCD — pull request #155 by kostrzewa (2012-09-24 09:30:26+00:00)
Title: additional OpenMP parallelization, Kahan summations, const qualifiers, MPI initialization

This pull-request collects:
* additional OpenMP parallelization of functions which (serially) account for few-percent level contributions to total runtime but blow up severely when thread-numbers are high (as on BG/Q)
* const qualifiers for the arguments of many often-used functions which seem to improve performance slightly on Intel (ICC) and BG/Q
* a new set of global arrays to accumulate the results of OpenMP parallelized Kahan summations
* bugfixes for thread-safety problems due to return values of functions: get_staples, exposu3, restoresu3, six_invert and six_det
* Kahan summations with OpenMP parallelization in many places
* correct MPI initialization when running in hybrid mode (MPI_Init_thread)
* compiler optimization options for BG/Q
* correct initialization of the random number generator with a user-defined seed

These changes have been tested with high statistics runs and the results look good, even though I feel that perhaps our error estimates are a bit low given the fluctuation of results with seed number.

http://www-zeuthen.desy.de/~kostrzew/expvals_more_openmp_1.pdf
http://www-zeuthen.desy.de/~kostrzew/expvals_more_openmp_2.pdf

No tests of the inverter have been carried out, but cg and bicg converge in the HMC.