Genre: Information/Explanation

GPU-Based Parallel Integration of Large Numbers of Independent ODE Systems ∗ arXiv : 1611 . 02274v1 [ cs . MS ] 6 Nov 2016 Kyle E. Niemeyer † School of Mechanical , Industrial , & Manufacturing Engineering , Oregon State University , OR , USA Chih-Jen Sung Department of Mechanical Engineering , University of Connecticut , Storrs , CT , USA 9 June 2014 Abstract The task of integrating a large number of independent ODE systems arises in various scientific and engineering areas . For nonstiff systems , common explicit integration algorithms can be used on GPUs , where individual GPU threads concurrently integrate independent ODEs with different initial conditions or parameters . One example is the fifth-order adaptive Runge – Kutta – Cash – Karp ( RKCK ) algorithm . In the case of stiff ODEs , standard explicit algorithms require impractically small time-step sizes for stability reasons , and implicit algorithms are therefore commonly used instead to allow larger time steps and reduce the computational expense . However , typical high-order implicit algorithms based on backwards differentiation formulae ( e.g. , VODE , LSODE ) involve complex logical flow that causes severe thread divergence when implemented on GPUs , limiting the performance . Therefore , alternate algorithms are needed . A GPUbased Runge – Kutta – Chebyshev ( RKC ) algorithm can handle moderate levels of stiffness and performs significantly faster than not only an equivalent CPU version but also a CPU-based implicit algorithm ( VODE ) based on results shown in the literature . In this chapter , we present the mathematical background , implementation details , and source code for the RKCK and RKC algorithms for use integrating large numbers of independent systems of ODEs on GPUs . In addition , brief performance comparisons are shown for each algorithm , demonstrating the potential benefit of moving to GPU-based ODE integrators . ∗ Final version published in Numerical Computations with GPUs ( 2014 ) Springer International Publishing , Ch. 8 , pp. 159 – 182 . V. Kindratenko ( Ed . ) . doi : 10 . 1007 / 978-3-319-06548-9_8 † Corresponding author ; Email : kyle.niemeyer@oregonstate.edu 1 Introduction In a number of science and engineering applications , researchers are faced with the solution of large numbers of independent systems of ordinary differential equations ( ODEs ) . One prominent example is simulation of reactive flows for modeling combustion [ 1 – 5 ] , atmospheric chemistry [ 6 , 7 ] , and groundwater transport [ 8 , 9 ] , where operator splitting [ 10 , 11 ] decouples the solution of the fluid transport ( e.g. , advection , diffusion ) and chemical kinetics terms . This results in large numbers of independent ODEs for conservation of chemical species masses , with one system for each spatial location . The solution of the aggregate of these ODEs consumes most of the total wall-clock time of such simulations , > 90 % in some cases . Simulations of electrical behavior in cardiac tissue use a similar operator splitting technique , which results in ODE systems for cell membrane dynamics [ 12 , 13 ] . Other areas that deal with solving many independent systems of ODEs include systems biology [ 14 , 15 ] and Monte Carlo methods for sensitivity analysis of ODEs [ 16 – 18 ] . In such problems , large numbers of the same governing ODEs with different initial conditions and / or input parameters must be integrated ; since each system is independent , the entire set of ODEs can be integrated concurrently . While performance can be improved by using parallel central processing unit ( CPU ) methods , this embarrassingly parallel problem is especially well-suited to acceleration with the thread-based parallelism of graphics processing units ( GPUs ) , as demonstrated for reactive-flow simulations [ 19 – 22 ] . In particular , Niemeyer and Sung [ 22 ] recently developed a GPU-based approach for the integration of moderately stiff chemical kinetics ODEs using explicit integration algorithms , using an adaptive fifth-order Runge – Kutta – Cash – Karp ( RKCK ) method for nonstiff cases and a stabilized second-order Runge – Kutta – Chebyshev ( RKC ) method for problems with greater stiffness . For large numbers of ODEs , they demonstrated that the GPU-based RKCK and RKC algorithms performed up to 126 and 65 times faster , respectively , than CPU versions of the same algorithm on a single CPU core . Furthermore , with moderate levels of stiffness , the GPU-based RKC offered a speedup factor of 57 compared to a conventional implicit algorithm executed in parallel on a six-core CPU . The specific acceleration demonstrated depending on the problem studied ( e.g. , the kinetic mechanism ) and number of ODEs considered . Due to the favorable performance of these methods , in this chapter we present the integration algorithms , associated GPU source code , and implementation details so that interested readers can apply them to more general applications ( e.g. , the areas described above ) . 2 Mathematical Background In this chapter , we represent a generic system of ODEs using d ~ y = f ~ ( t , ~ y ( t ) , ~ g ) , dt ( 1 ) where ~ y ( t ) is the vector of unknown dependent variables at some time t , f ~ is the right-hand-side vector function , and ~ g is a vector of constant parameters ( e.g. , pressure or temperature for chemical kinetics ) . The size of ~ y ( i.e. , the number of equations / unknowns ) is N. For the types of problems with which we are concerned here , a large number of ODE systems , Node , each given by Eq. ( 1 ) must be integrated independently from some time t = tn to tn + 1 , with different initial conditions ~ y ( tn ) and constant parameters ~ g for each system . The numerical approximation to the exact solution ~ y ( tn ) is ~ yn , and the step size for a given step is δtn = tn + 1 − tn . Nonstiff ODEs , or those with little stiffness , can be solved using explicit integration methods . Many such methods exist , and algorithms can be classified in general into Runge – Kutta and linear 1 i ai 1 2 3 4 5 6 1 5 3 10 3 5 1 7 8 j bij ci c ∗ i 37 378 2825 27648 250 621 125 594 18575 48384 13525 55296 277 14336 1 4 0 1 5 3 40 3 10 11 − 54 1631 55296 9 40 9 − 10 5 2 175 512 6 5 − 70 27 575 13824 35 27 44275 110592 253 4096 1 2 3 4 5 0 512 1771 0 Table 1 : Coefficients for the fifth-order Runge – Kutta – Cash – Karp method , adopted from Press et al. [ 26 ] . multistep methods , and also into explicit or implicit methods ; see Hairer and Wanner [ 23 ] for more details . Stiffness , a concept somewhat difficult to quantify , refers to the quality of an ODE making standard explicit methods perform poorly due to the requirement for unreasonably small time-step sizes – otherwise the solutions become unstable and oscillate wildly [ 24 ] . Traditionally , implicit integration algorithms such as those based on backwards difference formulas have been used to handle stiff equations , but these require expensive linear algebra operations on the Jacobian matrix ( e.g. , LU decomposition , matrix inversion ) . The complex logical flow of such operations results in highly divergent instructions across different initial conditions , making implicit algorithms unsuitable for operation on GPUs . In fact , Stone and Davis [ 21 ] implemented a traditional high-order implicit algorithm on GPUs , and found that it performed only slightly better than a multi-core CPU version of the same algorithm would . While implicit algorithms may be required for ODE systems with extreme levels of stiffness ( suggesting that new solutions need to be found for GPU acceleration of such problems ) , other options can be used for cases of little-to-moderate stiffness . Here , we describe two integration algorithms suitable for use solving many independent systems of ODEs on GPUs . 2.1 Runge – Kutta – Cash – Karp For nonstiff ODEs , an appropriate explicit algorithm is the fifth-order Runge – Kutta method developed by Cash and Karp [ 25 ] : the RKCK method . The RKCK method estimates the local truncation error using an embedded fourth-order method , by taking the difference between the fourth - and fifth-order solutions . It then uses this estimate to adaptively select the step size [ 26 ] . Using the terminology established above , the RKCK formulas – which also apply to any general 2 fifth-order Runge – Kutta method – are k ~ 1 = δtn f ~ ( tn , ~ yn , ~ g ) , ( 2 ) ~ k2 = δtn f ~ tn + a2 δtn , ~ yn + b21 ~ k1 , ~ g ( 3 ) , ~ k3 = δtn f ~ tn + a3 δtn , ~ yn + b31 ~ k1 + b32 ~ k2 , ~ g ( 4 ) , ~ k4 = δtn f ~ tn + a4 δtn , ~ yn + b41 ~ k1 + b42 ~ k2 + b43 ~ k3 , ~ g ( 5 ) , ~ k5 = δtn f ~ tn + a5 δtn , ~ yn + b51 ~ k1 + b52 ~ k2 + b53 ~ k3 + b54 ~ k4 , ~ g ~ k6 = δtn f ~ tn + a6 δtn , ~ yn + b61 ~ k1 + b62 ~ k2 + b63 ~ k3 + b64 ~ k4 + b65 ~ k5 , ~ g ~ yn + 1 = ~ yn + c1 ~ k1 + c2 ~ k2 + c3 ~ k3 + c4 ~ k4 + c5 ~ k5 + c6 ~ k6 , ∗ ~ yn + 1 = ~ yn + c ∗ 1 ~ k1 + c ∗ 2 ~ k2 + c ∗ 3 ~ k3 + c ∗ 4 ~ k4 + c ∗ 5 ~ k5 + c ∗ 6 ~ k6 , ( 6 ) ,, ( 7 ) ( 8 ) ( 9 ) ∗ where ~ yn + 1 is the fifth-order solution and ~ yn + 1 is the solution of the embedded fourth-order method . The RKCK coefficients are given in Table 1 . The local error ∆ n + 1 is estimated using the difference between the fourth - and fifth-order solutions : ∗ ∆ n + 1 = ~ yn + 1 − ~ yn + 1 = 6 X ( ci − c ∗ i ) ~ ki . ( 10 ) i = 1 Then , this error is compared against a desired accuracy , ∆ 0 , defined by ∆ 0 = ε | ~ yn | + δtn f ~ ( tn , ~ yn , ~ g ) + δ , ( 11 ) where ε is a tolerance level and δ represents a small value ( e.g. , 10 − 30 ) . When the estimated error rises above the desired accuracy ( ∆ n + 1 > ∆ 0 ) , the algorithm rejects the current step and calculates a smaller step size . Correspondingly , the algorithms accepts a step with error at or below the desired accuracy ( ∆ n + 1 ≤ ∆ 0 ) and calculates a new step size for the next step using ! 1 / 5 ∆ 0 , i if ∆ n + 1 ≤ ∆ 0 , or S δt max n ∆ n + 1 , i i δtnew = ! 1 / 4 ∆ 0 , i if ∆ n + 1 > ∆ 0 . S δtn max i ∆ n + 1 , i ( 12 ) Here , i represents the ith element of the related vector and S denotes a safety factor slightly smaller than unity ( e.g. , 0.9 ) . Equation ( 12 ) is used to calculate the next time step size both for an accepted step and also for a new , smaller step size in the case of a step rejection . In practice , step size decreases and increases are limited to factors of ten and five , respectively [ 26 ] . 2.2 Runge – Kutta – Chebyshev For ODEs with moderate levels of stiffness , one alternative to implicit algorithms is a stabilized explicit scheme such as the RKC method [ 27 – 32 ] . While the RKC method is explicit , it handles stiffness through additional stages – past the first two required for second-order accuracy – that extend its stability domain along the negative real axis of eigenvalues . Our RKC implementation is taken from Sommeijer et al. [ 31 ] and Verwer et al. [ 32 ] . Following the same terminology as above , the formulas for the second-order RKC are w ~ 0 = ~ yn , ( 13 ) 3 w ~ 1 = w ~ 0 + μ ̃ 1 δtn f ~ 0 , ( 14 ) w ~ j = ( 1 − μj − νj ) w ~ 0 + μj w ~ j − 1 + νj w ~ j − 2 + μ ̃ j δtn f ~ j − 1 + γ ̃ j δtn f ~ 0 , j = 2 , ... , s , ~ yn + 1 = w ~ s , ( 15 ) ( 16 ) where s is the total number of stages and w ~ j are internal vectors for the stages . The right-hand-side ~ derivative vector function , fj , is evaluated at each stage , such that f ~ j = f ~ ( tn + cj δtn , w ~ j , ~ g ) . The recursive nature of w ~ j allows the use of only five arrays for storage . The coefficients used in Eqs . ( 14 ) and ( 15 ) can be obtained analytically for any number of stages s ≥ 2 : μ ̃ 1 = b1 ω1 , γ ̃ j = − aj − 1 μ ̃ j , w0 = 1 + κ , s2 2bj ω0 , bj − 1 νj = − bj , bj − 2 b0 = b2 , b1 = 1 , ω0 μj = ω1 = Ts0 ( ω0 ) , Ts00 ( ω0 ) 2bj ω1 , bj − 1 Tj00 ( ω0 ) μ ̃ j = bj = 2 , Tj0 ( ω0 ) aj = 1 − bj Tj ( ω0 ) , ( 17 ) ( 18 ) ( 19 ) for j = 2 , ... , s , where κ ≥ 0 is the damping parameter ( e.g. , κ = 2 / 13 [ 31 , 32 ] ) . The Chebyshev polynomials of the first kind , Tj ( x ) with first and second derivatives Tj0 ( x ) and Tj00 ( x ) , respectively , are defined recursively as Tj ( x ) = 2xTj − 1 ( x ) − Tj − 2 ( x ) , j = 2 , ... , s , ( 20 ) where T0 ( x ) = 1 and T1 ( x ) = x . The cj used in the function evaluations are c1 = cj = c2 c2 , ≈ 0 T2 ( ω0 ) 4 Ts0 ( ω0 ) Tj00 ( ω0 ) Ts00 ( ω0 ) Tj0 ( ω0 ) ( 21 ) ≈ j2 − 1 , s2 − 1 2 ≤ j ≤ s − 1 , cs = 1 . ( 22 ) ( 23 ) As with the RKCK method in Sec. 2.1 , the RKC method can also be used with an adaptive time stepping method for error control , as given by Sommeijer et al. [ 31 ] . The error accrued in taking the step tn + 1 = tn + δtn and obtaining ~ yn + 1 is estimated using 4 2 ∆ n + 1 = ( ~ yn − ~ yn + 1 ) + δtn ( f ~ n + f ~ n + 1 ) . 5 5 ( 24 ) We then obtain the weighted RMS norm of error using this error estimate with absolute and relative tolerances : ∆ n + 1 k ∆ n + 1 kRMS = √ , ~ T N 2 ~ ~ T = A + R · max ( | ~ yn | , | ~ yn + 1 | ) , ( 25 ) ( 26 ) ~ is the vector of where N represents the number of unknown variables as defined previously , A absolute tolerances , and R is the relative tolerance . The norm k · k2 indicates the Euclidean or L2 norm . If k ∆ n + 1 kRMS ≤ 1 , the step is accepted ; otherwise , it is rejected and repeated using a 4 smaller step size . Finally , a new step size is calculated using the weighted RMS norm of error for the current and prior steps , as well as the associated step sizes , via δtn + 1 = min ( 10 , max ( 0 . 1 , f )) δtn , f = 0.8 1 / ( p + 1 ) k ∆ n kRMS 1 / ( p + 1 ) k ∆ n + 1 kRMS δtn δtn − 1 ( 27 ) 1 ! 1 / ( p + 1 ) , k ∆ n kRMS ( 28 ) where p = 2 , the order of the algorithm . We use Eq. ( 27 ) with a modified relation to calculate a new step size for a step rejection : 0.8 f = . ( 29 ) 1 / ( p + 1 ) k ∆ n kRMS Determining the initial time step size requires special consideration . First , the algorithm takes a tentative integration step , using the inverse of the spectral radius σ – the magnitude of the largest eigenvalue – of the Jacobian matrix as the step size . Then , after estimating the error associated with this tentative step , it calculates a new step size following a similar procedure to that given in Eqs . ( 27 ) and ( 28 ) : 1 , σ ∆ 0 = δt0 f ~ ( t0 + δt0 , ~ y0 + δt0 f ~ ( t0 , ~ y0 )) − f ~ ( t0 , ~ y0 ) , δt0 = δt1 = 0.1 δt0 1 / 2 ( 30 ) ( 31 ) ( 32 ) , k ∆ 0 kRMS ~ 0 kRMS is evaluated in the same manner as k ∆ ~ n + 1 kRMS using Eq. ( 25 ) . where k ∆ After selecting the optimal time step size to control local error , the algorithm next determines the optimal number of RKC stages in order to remain stable . Due to stiffness , too few stages could lead to instability ; in contrast , using more stages than required would add unnecessary computational effort . The number of stages is determined using the spectral radius and time step size : s = 1 + p 1 + 1.54 δtn σ , ( 33 ) as suggested by Sommeijer et al. [ 31 ] , where the value 1.54 is related to the stability boundary of the algorithm . Note that s may vary between time steps due to a changing spectral radius and time step size . We recommend using a nonlinear power method [ 31 ] to calculate the spectral radius with our RKC implementation ; this choice costs an additional vector to store the computed eigenvector , but avoids storing or calculating the Jacobian matrix directly . Following Sommeijer et al. [ 31 ] , our RKC implementation estimates the spectral radius every 25 ( internal ) steps or after a step rejection . 3 Source Code Next , we provide implementation details and source code for the GPU versions of the algorithms described above . The number of unknowns ( and corresponding equations ) N is represented with the variable NEQN , and the number of ODE systems Node is defined as numODE in the following code . In order for the GPU algorithms to offer a performance increase over CPU algorithms , Node should be relatively large . Although the exact number of ODEs where the GPU algorithm becomes faster than its CPU equivalent is problem dependent , Niemeyer and Sung [ 22 ] showed that a GPU implementation of the RKCK algorithm for chemical kinetics outperforms an equivalent serial CPU 5 version for as little as 128 ODE systems . All operations are given here in double precision , although depending on the particular needs of the specific application single-precision calculations may be preferable to reduce the computational expense . In order to take advantage of global memory coalescing on the GPU , we recommend storing the the set of dependent variable vectors ~ yi , where i = 1 , ... , Node , in a single one-dimensional array , where variables corresponding to the same unknown for consecutive systems sit adjacent in memory . ~ is a two-dimensional matrix with Node rows and N columns , where the ith row In other words , if Y ~ should be stored in memory as a one-dimensional array in contains the unknown vector ~ yi , then Y column-major ordering . This ensures that adjacent GPU threads in the same warp access adjacent global memory locations when reading or writing equivalent array elements . See Kirk and Hwu [ 33 ] or Jang et al. [ 34 ] for more details and examples on global memory coalescing . The following code snippet shows the proper loading of a host array yHost from an arbitrary array y that contains initial conditions for all ODEs : 1 2 double * yHost ; yHost = ( double * ) malloc ( numODE * NEQN * sizeof ( double )) ; 3 4 5 6 7 8 for ( int i = 0 ; i < numODE ; ++ i ) { for ( int j = 0 ; j < NEQN ; ++ j ) { yHost [ i + numODE * j ] = y [ i ] [ j ] ; } } A similar procedure should be used for the constant parameter vector ~ g if needed . Next , the GPU global memory arrays should be declared and initialized , and the block / grid dimensions set up using 1 2 double * yDevice ; cudaMalloc (( void ** ) & yDevice , numODE * NEQN * sizeof ( double )) ; 3 4 5 6 7 8 9 10 11 12 13 14 15 int blockSize ; if ( numODE < 4194304 ) { blockSize = 64 ; } else if ( numODE < 8388608 ) { blockSize = 128 ; } else if ( numODE < 16777216 ) { blockSize = 256 ; } else { blockSize = 512 ; } dim3 dimBlock ( blockSize , 1 ) ; dim3 dimGrid ( numODE / dimBlock . x , 1 ) ; Here , we use simple one-dimensional block and grid dimensions ; reshaping the grid should not affect performance , but can be done for convenience . We chose 64 as the block size for problems with less than 4,194,304 ODEs based on experience for chemical kinetics problems [ 22 ] . The size should remain a power of two , but different block sizes may be optimal for other problems . The final step is to set up the ODE integration loop and kernel function execution . The integration driver kernel , to be described shortly , will perform internal sub-stepping as necessary to reach the specified end time . Depending on the objectives , there are various ways to approach this : • If only the final integrated results are needed , then a single GPU kernel can be invoked . • If intermediate integration results are needed , then an acceptable outer step size over which results will be spaced should be chosen , and the GPU kernel should be invoked inside a loop . 6 We will leave the code as general as possible by following the second approach , although modifications should be made depending on the desired functionality . In both cases , the global memory array holding the variables to be integrated needs to be transferred to the GPU before , and from the GPU after , each kernel invocation . 1 2 3 // set initial time double t = t0 ; double tNext = t + h ; 4 5 6 7 while ( t < tEnd ) { // transfer memory to GPU cudaMemcpy ( yDevice , yHost , numODE * NEQN * sizeof ( double ) , c uda Me mc py Hos tT oD ev i c e ) ; 8 intDriver <<< dimGrid , dimBlock > > > ( t , tNext , numODE , gDevice , yDevice ) ; 9 10 // transfer memory back to CPU cudaMemcpy ( yHost , yDevice , numODE * NEQN * sizeof ( double ) , c uda Me mc py Dev ic eT oH o s t ) ; 11 12 13 t = tNext ; tNext + = h ; 14 15 16 } 17 18 19 cudaFree ( gDevice ) ; cudaFree ( yDevice ) ; Here , t0 refers to the initial time , tEnd the desired final time , and h the outer step size . In the current form , each outer integration step performed by the GPU will be a " restart " integration , meaning no information about previous steps ( e.g. , error estimates , step sizes ) will be used to assist the startup . This is necessary in certain applications such as reactive-flow simulations ( and other simulation methods that use operator splitting ) , where after each outer step integration results are combined with changes due to fluid transport , thereby invalidating stored integration information . However , where possible , better performance may be obtained by transferring the appropriate data from the GPU and used in the next overall integration step . The next code snippet contains the general integration driver kernel , suitable for either algorithm : 1 2 3 4 5 __global__ void intDriver ( const double t , const double tEnd , const int numODE , const double * gGlobal , double * yGlobal ) { // unique thread ID , based on local ID in block and block ID int tid = threadIdx . x + ( blockDim . x * blockIdx . x ) ; 6 7 8 // ensure thread within limit if ( tid < numODE ) { 9 10 11 // local array with initial values double yLocal [ NEQN ] ; 12 13 14 // constant parameter ( s ) double gLocal = gGlobal [ tid ] ; 15 16 17 18 19 // load local array with initial values from global array for ( int i = 0 ; i < NEQN ; ++ i ) { yLocal [ i ] = yGlobal [ tid + numODE * i ] ; } 20 21 22 // call integrator for one time step integrator ( t , tEnd , yLocal , gGlobal ) ; 23 7 // update global array with integrated values for ( int i = 0 ; i < NEQN ; ++ i ) { yGlobal [ tid + numODE * i ] = yLocal [ i ] ; } } 24 25 26 27 28 29 } The function integrator should be replaced with rkckDriver or rkcDriver ( given below ) depending on the desired integration algorithm . 3.1 RKCK Code In the following , the source code for the RKCK driver device function is given in functional snippets . First , the minimum and maximum allowable time step sizes are defined , and the initial step size is set as half the integration time . 1 2 3 __device__ void rkckDriver ( double t , const double tEnd , const double g , double * y ) { 4 5 6 7 // maximum and minimum allowable step sizes const double hMax = fabs ( tEnd - t ) ; const double hMin = 1.0 e -20 ; 8 9 10 // initial step size double h = 0.5 * fabs ( tEnd - t ) ; Then , inside the time integration loop , the algorithm takes a trial integration step and estimates the error of that step . 1 2 // integrate until specified end time while ( t < tEnd ) { 3 4 5 // limit step size based on remaining time h = fmin ( tEnd - t , h ) ; 6 7 8 // y and error vectors temporary until error determined double yTemp [ NEQN ] , yErr [ NEQN ] ; 9 10 11 12 // evaluate derivative double F [ NEQN ] ; dydt ( t , y , g , F ) ; 13 14 15 // take a trial step rkckStep ( t , y , g , F , h , yTemp , yErr ) ; 16 17 18 19 20 21 // calculate error double err = 0.0 ; int nanFlag = 0 ; for ( int i = 0 ; i < NEQN ; ++ i ) { if ( isnan ( yErr [ i ] ) ) nanFlag = 1 ; 22 23 24 25 err = fmax ( err , fabs ( yErr [ i ] / ( fabs ( y [ i ] ) + fabs ( h * F [ i ] ) + TINY ))) ; } err / = eps ; If the error is too large , the step size is decreased and the step retaken ; otherwise , the algorithm accepts the step and calculates the next step size , then repeats the process . 8 // check if error too large if (( err > 1.0 ) || isnan ( err ) || ( nanFlag == 1 )) { // step failed , error too large if ( isnan ( err ) || ( nanFlag == 1 )) { h * = P1 ; } else { h = fmax ( SAFETY * h * pow ( err , PSHRNK ) , P1 * h ) ; } 1 2 3 4 5 6 7 8 9 } else { // step accepted t + = h ; 10 11 12 13 if ( err > ERRCON ) { h = SAFETY * h * pow ( err , PGROW ) ; } else { h * = 5.0 ; } 14 15 16 17 18 19 // ensure step size is bounded h = fmax ( hMin , fmin ( hMax , h )) ; 20 21 22 for ( int i = 0 ; i < NEQN ; ++ i ) y [ i ] = yTemp [ i ] ; 23 24 } 25 } 26 27 } The device function dydt evaluates the derivative function F for the particular problem as in Eq. ( 1 ) using the input time t , vector of dependent variables y , and constant parameter ( s ) g . The device function rkcStep , not given here , takes a single integration step using Eqs . ( 2 ) – ( 9 ) , returning the vector of integrated values yTemp as well as the error vector yErr . A number of constants were used in this function , given here : 1 2 3 4 5 6 7 # define UROUND ( 2 . 22 e -16 ) # define SAFETY 0.9 # define PGROW ( - 0 . 2 ) # define PSHRNK ( - 0 . 25 ) # define ERRCON ( 1 . 89 e -4 ) # define TINY ( 1 . 0 e -30 ) const double eps = 1.0 e -10 ; 3.2 RKC Code The RKC driver algorithm is next given . For this algorithm , the number of stages must be determined at each step to handle local stiffness ; to avoid excess computation , a maximum number of stages is first set . In addition , minimum and maximum allowable time step sizes are defined . 1 2 3 4 __device__ void rkcDriver ( double t , const double tEnd , const double g , double * y ) { // number of steps int numStep = 0 ; 5 6 7 // maximum allowable number of RKC stages int mMax = ( int ) ( round ( sqrt ( relTol / ( 10 . 0 * UROUND )))) ; 8 9 10 // RKC needs at least two stages for second - order accuracy if ( mMax < 2 ) mMax = 2 ; 9 11 12 13 // maximum allowable step size const double stepSizeMax = fabs ( tEnd - t ) ; 14 15 16 // minimum allowable step size double stepSizeMin = 10.0 * UROUND * fmax ( fabs ( t ) , stepSizeMax ) ; Then , the algorithm evaluates the derivative using the initial conditions for use as the initial eigenvector estimate for the spectral radius calculation . The calculated eigenvectors are stored and used as initial guesses in later steps . 1 2 3 4 // internal y vector double y_n [ NEQN ] ; for ( int i = 0 ; i < NEQN ; ++ i ) y_n [ i ] = y [ i ] ; 5 6 7 8 // calculate F_n for initial y double F_n [ NEQN ] ; dydt ( t , y_n , g , F_n ) ; 9 10 11 // internal work vector double work [ 4 + NEQN ] ; 12 13 14 15 16 17 18 // load initial estimate for eigenvector if ( work [ 2 ] < UROUND ) { for ( int i = 0 ; i < NEQN ; ++ i ) { work [ 4 + i ] = F_n [ i ] ; } } Inside the time integration loop , the algorithm calculates the spectral radius for the first step – which it next uses to determine the initial step size – and every 25 steps thereafter . 1 2 3 // perform internal sub - stepping while ( t < tEnd ) { double tempArr [ NEQN ] , tempArr2 [ NEQN ] , err ; 4 5 6 // if last step , limit step size if (( 1 . 1 * work [ 2 ] ) > = fabs ( tEnd - t )) work [ 2 ] = fabs ( tEnd - t ) ; 7 8 9 10 11 // estimate Jacobian spectral radius if 25 steps passed if (( numStep % 25 ) == 0 ) { work [ 3 ] = rkcSpecRad ( t , y_n , g , F_n , stepSizeMax , & work [ 4 ] , tempArr2 ) ; } For the initial step , a trial step is taken using the inverse of the spectral radius as the step size ; the resulting error is used to determine an appropriate step size that satisfies error control . 1 2 3 4 // if this is initial step if ( work [ 2 ] < UROUND ) { // estimate first time step work [ 2 ] = stepSizeMax ; 5 6 7 if (( work [ 3 ] * work [ 2 ] ) > 1.0 ) work [ 2 ] = 1.0 / work [ 3 ] ; work [ 2 ] = fmax ( work [ 2 ] , stepSizeMin ) ; 8 9 10 11 12 for ( int i = 0 ; i < NEQN ; ++ i ) { temp_arr [ i ] = y_n [ i ] + ( work [ 2 ] * F_n [ i ] ) ; } dydt ( t + work [ 2 ] , tempArr , g , tempArr2 ) ; 10 13 err = 0.0 ; for ( int i = 0 ; i < NEQN ; ++ i ) { double est = ( tempArr2 [ i ] - F_n [ i ] ) / ( absTol + relTol * fabs ( y_n [ i ] )) ; err + = est * est ; } err = work [ 2 ] * sqrt ( err / NEQN ) ; 14 15 16 17 18 19 20 if (( P1 * work [ 2 ] ) < ( stepSizeMax * sqrt ( err ))) { work [ 2 ] = fmax ( P1 * work [ 2 ] / sqrt ( err ) , stepSizeMin ) ; } else { work [ 2 ] = stepSizeMax ; } 21 22 23 24 25 26 } For all steps following the first , the value stored in work [ 2 ] is used for the time step size . Next , the number of stages is determined using the spectral radius and current time step size , and a tentative integration step performed . 1 2 // calculate number of steps int m = 1 + ( int ) ( sqrt ( 1 . 54 * work [ 2 ] * work [ 3 ] + 1.0 )) ; 3 4 5 6 7 8 // modify step size based on stages if ( m > mMax ) { m = mMax ; work [ 2 ] = (( double ) ( m * m - 1 ) ) / ( 1 . 54 * work [ 3 ] ) ; } 9 10 11 // perform tentative time step rkcStep ( t , y_n , g , F_n , work [ 2 ] , m , y ) ; The algorithm then estimates the error of that step . 1 2 // calculate derivative F_np1 with tentative y_np1 dydt ( t + work [ 2 ] , y , g , tempArr ) ; 3 4 5 6 7 8 9 10 11 // estimate error err = 0.0 ; for ( int i = 0 ; i < NEQN ; ++ i ) { double est = 0.8 * ( y_n [ i ] - y [ i ] ) + 0.4 * work [ 2 ] * ( F_n [ i ] + tempArr [ i ] ) ; est / = ( absTol + relTol * fmax ( fabs ( y [ i ] ) , fabs ( y_n [ i ] ))) ; err + = est * est ; } err = sqrt ( err / (( double ) N )) ; Based on the error magnitude , the algorithm determines whether to accept the step and proceed to the next step or decrease the step size and repeat the current step . 1 2 3 4 5 // check value of error if ( err > 1.0 ) { // error too large , step is rejected // select smaller step size work [ 2 ] = 0.8 * work [ 2 ] / ( pow ( err , ( 1 . 0 / 3.0 ))) ; 6 7 8 9 10 11 12 // reevaluate spectral radius work [ 3 ] = rkcSpecRad ( t , y_n , g , F_n , stepSizeMax , & work [ 4 ] , tempArr2 ) ; } else { // step accepted t + = work [ 2 ] ; numStep ++ ; 11 Finally , for an accepted step , the current step size and error are stored and the next step size is calculated . double fac = 10.0 ; double temp1 , temp2 ; 1 2 3 if ( work [ 1 ] < UROUND ) { temp2 = pow ( err , ( 1 . 0 / 3.0 )) ; if ( 0 . 8 < ( fac * temp2 )) fac = 0.8 / temp2 ; } else { temp1 = 0.8 * work [ 2 ] * pow ( work [ 0 ] , ( 1 . 0 / 3.0 )) ; temp2 = work [ 1 ] * pow ( err , ( 2 . 0 / 3.0 )) ; if ( temp1 < ( fac * temp2 )) fac = temp1 / temp2 ; } 4 5 6 7 8 9 10 11 12 // set " old " values to those for current time step work [ 0 ] = err ; work [ 1 ] = work [ 2 ] ; 13 14 15 16 for ( int i = 0 ; i < NEQN ; ++ i ) { y_n [ i ] = y [ i ] ; F_n [ i ] = tempArr [ i ] ; } 17 18 19 20 21 work [ 2 ] * = fmax ( P1 , fac ) ; work [ 2 ] = fmax ( stepSizeMin , fmin ( stepSizeMax , work [ 2 ] )) ; 22 23 } 24 } 25 26 } As before , we do not provide the RKC integration step device function rkcStep , which evaluates Eqs . ( 13 ) – ( 16 ) . The absolute and relative tolerances absTol and relTol are set as defined constants , e.g. , : 1 2 const double abs_tol = 1.0 e -10 ; const double rel_tol = 1.0 e -6 ; Note that these may be modified to more stringent tolerances if desired . The constant UROUND is defined the same as in the RKCK code above . The local work array work contains , in element order : 0 the previous step error estimation , 1 previous time step , 2 current time step , 3 spectral radius , and 4 vector of eigenvalues ( of size N ) . The device function rkcSpecRad returns the spectral radius , the largest magnitude eigenvalue ; various methods may be used for this purpose depending on the case . We provide GPU source code for a nonlinear power method adopted from Sommeijer et al. [ 31 ] that may be used for general applications in the Appendix . 12 4 Performance Results We tested the performance of the GPU-based RKCK and RKC integration algorithms using two ODE test cases , ranging the number of ODE systems from 101 to 105 . For both cases , all calculations were performed in double precision using a single GPU and single CPU ; we compared the performance of the GPU algorithm against serial CPU calculations as well as parallelized CPU performance – via OpenMP [ 35 ] – on four cores . We performed the GPU calculations using an NVIDIA Tesla c2075 GPU with 6 GB of global memory , and an Intel Xeon X5650 CPU , running at 2.67 GHz with 256 kB of L2 cache memory per core and 12 MB of L3 cache memory , served as the host processor both for the GPU calculations and the CPU single - and four-core OpenMP calculations . We used the GNU Compiler Collection ( gcc ) version 4.6.2 ( with the compiler options " - O3 - ffast-math - std = c99 - m64 " ) to compile the CPU programs and the CUDA 5.5 compiler nvcc version 5.5.0 ( " - O3 - arch = sm_20 - m64 " ) to compile the GPU versions . We set the GPU to persistence mode , but also used the cudaSetDevice ( ) to hide any further device initialization delay in the CUDA implementations prior to the timing . The integration algorithms take as input initial conditions and a global time step , performing internal sub-stepping as necessary . The computational wall-clock times reported represent the average over ten global time steps , which for the GPU versions includes the overhead required for transmitting data between the CPU and GPU before and after each global step . The integrator restarts at each global time step , not storing any data from the previous step – although any substepping performed by the algorithm within these larger steps does benefit from retained information from prior sub-steps . Interested readers should refer to Niemeyer and Sung [ 22 ] for more detailed performance evaluations of these algorithms in the context of chemical kinetics problems . 4.1 RKCK Results We used the Pleiades ODE test problem ( PLEI ) of Hairer et al. [ 23 , 36 ] to test the GPU - and CPU-based versions of the RKCK integrator . This nonstiff test case originates from a celestial mechanisms problem tracking the coordinates of seven stars ; it consists of a set of 14 second-order ODEs based on Newtonian gravitational forces , in the form ~ x ~ z00 = ~ y ! 00 ( 1 ) x00i = fi f ~ ( 1 ) ( ~ x , ~ y ) , f ~ ( 2 ) ( ~ x , ~ y ) ! = ( ~ x , ~ y ) = X ~ z ∈ < 14 , ( 34 ) mj ( xj − xi ) / rij , ( 35 ) mj ( yj − yi ) / rij , ( 36 ) 3 / 2 ( 37 ) j6 = i yi00 = ( 2 ) fi ( ~ x , ~ y ) = X j6 = i rij = ( xi − xj ) 2 + ( yi − yj ) , i , j = 1 , ... , 7 , where ( xi , yi ) and mi = i are the coordinates and mass of the ith star , respectively . This secondorder system can be converted into a system of 28 first-order ODEs of the form by defining w ~ = ~ z0 , such that ! 0 !! ~ z w ~ ~ z = ~ , ∈ < 28 . ( 38 ) w ~ f ( ~ z ) w ~ While the original problem offers specific initial conditions for a single ODE system , here we consider a large number of ODEs with the initial conditions randomly perturbed by a small factor to 13 4 Speedup factor of RKCK-GPU 3.5 3 2.5 2 1.5 1 0.5 0 103 RKCK-GPU vs. RKCK-CPU RKCK-GPU vs. RKCK-CPU × 4 104 105 Number of independent ODE systems Figure 1 : Speedup factors offered by GPU-based explicit RKCK integration algorithm over singleand four-core CPU-based versions for Pleiades ODE problem . Note that the horizontal axis is displayed in logarithmic scale . 14 emulate a sensitivity analysis . We integrated the ODE systems from t = 0 s to 1.0 s using 1.0 × 10 − 1 s as the global time step size . We set the RKCK tolerance ε ( eps in the code ) to 1.0 × 10 − 10 . Figure 1 shows the speedup factors , measured as the ratio of computational times per step , offered by the GPU-based RKCK algorithm over the baseline CPU version for both serial and four-core parallel operation , for numbers of ODE systems ranging from 1024 to 262,144 . The GPU-based algorithm ran faster than the serial and parallel CPU-based algorithms for Node larger than 4096 and 8192 , respectively . For the current problem , at best the GPU offered a speedup factors of nearly four and two over the serial and four-core CPU implementations , respectively . The non-smooth performance scaling resulted from the randomly perturbed initial conditions . Note that since each ODE system used randomly perturbed initial conditions , adjacent threads in the GPU implementation handled potentially extremely different initial condition values , resulting in thread divergence due to varying internal time step sizes . Therefore , the results shown here represent a worst-case GPU algorithm performance , particularly compared to applications involving operator-split reactive-flow codes where adjacent threads / ODE systems correspond to neighboring spatial locations . In such situations , initial conditions would be more similar and therefore follow similar instruction pathways . In either case , GPU-based integration algorithms offer performance benefits over the baseline CPU versions . See Niemeyer and Sung [ 22 ] for more discussion on this topic . Furthermore , the current problem involved a relatively simple system of ODEs , limiting the calculations performed on the GPU between the memory transfers before and after each integration step . ODE systems with more complex derivative functions would saturate the GPU with operations , increasing performance . For example , the RKCK algorithm demonstrated by Niemeyer and Sung [ 22 ] performed up to 126 times faster on a GPU than on a serial CPU , integrating a chemical kinetics ODE system with nine species participating in 38 reaction steps – requiring significantly more floating-point calculations than the case studied here . 4.2 RKC Results To demonstrate the performance of the GPU-based RKC algorithm , we used a chemical kinetics problem : the ODE system describing the constant-volume autoignition of ethanol ( C2 H5 OH ) . We implemented the reaction mechanism of Marinov [ 37 ] to describe the oxidation of ethanol , with 57 species participating in 766 irreversible reaction steps . The governing ODE system contained 58 ~ : equations : one for temperature T and the rest for species mass fractions Y d ~ y = dt dYNsp dT dY1 ,, ... , dt dt dt | , ( 39 ) Nsp dT 1 X = − ei ωi Wi , dt ρcv i = 1 dYi Wi ωi = , i = 1 , ... , Nsp , dt ρ ωi = NX reac Nsp 00 νij − 0 νij j = 1 kj Y ν0 Ck kj , ( 40 ) ( 41 ) ( 42 ) k = 1 where ρ indicates the density , cv the mass-averaged constant-volume specific heat , ei the internal 00 and ν 0 the forward and energy of the ith species , Wi the molecular weight of the ith species , νij ij reverse stoichiometric coefficients for the ith species in reaction j , Ck the molar concentration of the kth species , and Nsp and Nreac are the numbers of species and reactions , respectively . For a 15 Speedup factor of RKCK-GPU 100 10 1 0.1 RKC-GPU vs. RKC-CPU RKC-GPU vs. RKC-CPU × 4 102 103 104 Number of independent ODE systems 105 Figure 2 : Speedup factors offered by GPU-based explicit RKC integration algorithm over singleand four-core CPU-based versions for pollution ODE problem . Note that both axes are displayed in logarithmic scale . reaction j without pressure dependence , the rate coefficient kj is given in Arrhenius form by kj = Aj T βj − Ej exp RT , ( 43 ) where R is the universal gas constant , Aj the pre-exponential coefficient , βj the temperature exponent , and Ej the activation energy . Note that reactions can be pressure-dependent ( see , e.g. , Law [ 38 ] for examples of various pressure-dependence formulations ) ; these were also considered in the current implementation . This problem is moderately stiff using a time step size of δt = 1.0 × 10 − 6 s for 10 global time steps . In this case , we generated initial conditions for the set of ODE systems by sampling the solutions obtained from constant-pressure homogeneous ignition simulations , initiated at 1600 K , 1 atm , and an equivalence ratio of one1 . We assigned these initial conditions sequentially , such that adjacent threads in the GPU implementation contained data from consecutive time steps in the sample – and therefore such threads handled the integration of similar conditions , emulating adjacent spatial locations in an operator-split reactive-flow simulation . Figure 2 shows the speedup factors offered by the GPU-based RKC algorithm over the baseline CPU version for both serial and four-core parallel operation , for numbers of ODE systems ranging from 64 to 16,384 . In this case , the GPU-accelerated code ran faster than the serial CPU version for the entire range of ODE system sizes considered , while it offered better performance than the four-core parallel CPU version for 256 ODEs and higher . At best , the GPU-based RKC algorithm 1 An equivalence ratio of one indicates the mixture of fuel and oxidizer set to an appropriate ratio for complete combustion . 16 ran nearly 64 and 17 times faster than the serial and four-core CPU implementations , respectively . The discontinuity in speedup seen in Fig. 2 corresponded to the inclusion of conditions leading to greater stiffness . 5 Conclusions In this chapter , we presented two explicit algorithms appropriate for integrating large numbers of independent ODE systems on GPUs . Specifically , we proposed the fifth-order adaptive Runge – Kutta – Cash – Karp ( RKCK ) method for nonstiff problems and the stabilized second-order adaptive Runge – Kutta – Chebyshev ( RKC ) method for problems with moderate levels of stiffness . Source code and implementation details were presented to ease the adoption of such methods , and performance comparison results were presented for each method . The examples shown here served to demonstrate the potential of GPU acceleration where many independent systems of ODEs need to be integrated ; in the case of the RKC algorithm , we demonstrated more than an order of magnitude performance increase over an equivalent parallel CPU code running on four cores . The types of scientific and engineering problems dealing with large numbers of ODEs – in particular , reactive-flow models that rely on operator splitting – can benefit significantly from GPU acceleration ; interested readers can directly implement the algorithms presented here to such ends , or use them as the beginnings for their own solution . Acknowledgements This work was supported by the US Department of Defense through the National Defense Science and Engineering Graduate Fellowship program , the National Science Foundation Graduate Research Fellowship under grant number DGE-0951783 , and the Combustion Energy Frontier Research Center – an Energy Frontier Research Center funded by the US Department of Energy , Office of Science , Office of Basic Energy Sciences under award number DE-SC0001198 . Appendix Various methods may be used to calculate the spectral radius , including the Gershgorin circle theorem [ 39 , 40 ] that provides an upper-bound estimate . Here , we provide a function based on a nonlinear power method [ 31 ] . 1 2 3 4 __device__ double rkcSpecRad ( const double t , const double * y , const double g , const double * F , const double hMax , double * v , double * Fv ) { // maximum number of iterations const int itmax = 50 ; 5 6 double small = 1.0 / hmax ; 7 8 9 10 11 12 13 14 15 double nrm1 = 0.0 ; double nrm2 = 0.0 ; for ( int i = 0 ; i < NEQN ; ++ i ) { nrm1 + = ( y [ i ] * y [ i ] ) ; nrm2 + = ( v [ i ] * v [ i ] ) ; } nrm1 = sqrt ( nrm1 ) ; nrm2 = sqrt ( nrm2 ) ; 17 16 double dynrm ; if (( nrm1 ! = 0.0 ) && ( nrm2 ! = 0.0 )) { dynrm = nrm1 * sqrt ( UROUND ) ; for ( int i = 0 ; i < NEQN ; ++ i ) { v [ i ] = y [ i ] + v [ i ] * ( dynrm / nrm2 ) ; } } else if ( nrm1 ! = 0.0 ) { dynrm = nrm1 * sqrt ( UROUND ) ; for ( int i = 0 ; i < NEQN ; ++ i ) { v [ i ] = y [ i ] * ( 1 . 0 + sqrt ( UROUND )) ; } } else if ( nrm2 ! = 0.0 ) { dynrm = UROUND ; for ( int i = 0 ; i < NEQN ; ++ i ) { v [ i ] * = ( dynrm / nrm2 ) ; } } else { dynrm = UROUND ; for ( int i = 0 ; i < NEQN ; ++ i ) { v [ i ] = UROUND ; } } 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 // now iterate using nonlinear power method double sigma = 0.0 ; for ( int iter = 1 ; iter < = itmax ; ++ iter ) { 40 41 42 43 dydt ( t , pr , v , Fv ) ; 44 45 nrm1 = 0.0 ; for ( int i = 0 ; i < NEQN ; ++ i ) { nrm1 + = (( Fv [ i ] - F [ i ] ) * ( Fv [ i ] - F [ i ] )) ; } nrm1 = sqrt ( nrm1 ) ; nrm2 = sigma ; sigma = nrm1 / dynrm ; 46 47 48 49 50 51 52 53 nrm2 = fabs ( sigma - nrm2 ) / sigma ; if (( iter > = 2 ) && ( fabs ( sigma - nrm2 ) < = ( fmax ( sigma , small ) * 0.01 ))) { for ( int i = 0 ; i < NEQN ; ++ i ) { v [ i ] - = y [ i ] ; } return ( 1 . 2 * sigma ) ; } 54 55 56 57 58 59 60 61 if ( nrm1 ! = 0.0 ) { for ( int i = 0 ; i < NEQN ; ++ i ) { v [ i ] = y [ i ] + (( Fv [ i ] - F [ i ] ) * ( dynrm / nrm1 )) ; } } else { int ind = ( iter % NEQN ) ; v [ ind ] = y [ ind ] - ( v [ ind ] - y [ ind ] ) ; } 62 63 64 65 66 67 68 69 } return ( 1 . 2 * sigma ) ; 70 71 72 } 18 References [ 1 ] O. M. Knio , H. N. Najm , and P. S. Wyckoff , J. Comput . Phys . 154 , 428 ( 1999 ) . [ 2 ] M. S. Day and J. B. Bell , Combust . Theor . Model . 4 , 535 ( 2000 ) . [ 3 ] E. S. Oran and J. P. Boris , Numerical Simulation of Reactive Flow , 2nd ed . ( Cambridge University Press , 2001 ) . [ 4 ] D. Schwer , P. Lu , W. H. Green , and V. Semiao , Combust . Theor . Model . 7 , 383 ( 2003 ) . [ 5 ] Z. Ren and S. B. Pope , J. Comput . Phys . 227 , 8165 ( 2008 ) . [ 6 ] V. Alexandrov , A. Sameh , Y. Siddique , and Z. Zlatev , Environ . Monit . Assess . 2 , 365 ( 1997 ) . [ 7 ] J. Kim and S. Y. Cho , Atmos . Environ . 31 , 2215 ( 1997 ) . [ 8 ] D. Barry , C. Miller , P. Culligan , and K. Bajracharya , Math . Comput . Simulat . 43 , 331 ( 1997 ) . [ 9 ] D. Barry , K. Bajracharya , M. Crapper , H. Prommer , and C. Cunningham , Math . Comput . Simulat . 53 , 113 ( 2000 ) . [ 10 ] G. Strang , SIAM J. Numer . Anal . 5 , 506 ( 1968 ) . [ 11 ] B. Sportisse , J. Comput . Phys . 161 , 140 ( 2000 ) . [ 12 ] J. Sundnes et al. , Ann Biomed Eng 34 , 1088 ( 2006 ) . [ 13 ] V. K. Nimmagadda , A. Akoglu , S. Hariri , and T. Moukabary , J Supercomput 59 , 1360 ( 2011 ) . [ 14 ] L. Dematte and D. Prandi , Brief . Bioinform . 11 , 323 ( 2010 ) . [ 15 ] Y. Zhou , J. Liepe , X. Sheng , M. P. H. Stumpf , and C. Barnes , Bioinformatics 27 , 874 ( 2011 ) . [ 16 ] J. Helton and F. Davis , Reliab . Eng . Syst . Safe . 81 , 23 ( 2003 ) . [ 17 ] S. Marino , I. B. Hogue , C. J. Ray , and D. E. Kirschner , J. Theor . Biol . 254 , 178 ( 2008 ) . [ 18 ] C. Kühn et al. , BMC Syst . Biol . 3 , 83 ( 2009 ) . [ 19 ] K. E. Niemeyer , C. J. Sung , C. G. Fotache , and J. C. Lee , Turbulence-chemistry closure method using graphics processing units : a preliminary test , in 7th Fall Technical Meeting of the Eastern States Section of the Combustion Institute , Storrs , CT , 2011 . [ 20 ] Y. Shi , W. H. Green , H. Wong , and O. O. Oluwole , Combust . Flame 159 , 2388 ( 2012 ) . [ 21 ] C. P. Stone and R. L. Davis , J Propul Power 29 , 764 ( 2013 ) . [ 22 ] K. E. Niemeyer and C. J. Sung , J. Comput . Phys . 256 , 854 ( 2014 ) . [ 23 ] E. Hairer , G. Wanner , and S. P. Nørsett , Solving Ordinary Differential Equations I : Nonstiff Problems , Springer Series in Computational Mathematics Vol. 8 , 2nd ed . ( Springer-Verlag Berlin Heidelberg , Berlin Heidelberg , 1993 ) . [ 24 ] E. Hairer and G. Wanner , Solving Ordinary Differential Equations II : Stiff and DifferentialAlgebraic Problems , Springer Series in Computational Mathematics Vol. 14 , 2nd ed . ( SpringerVerlag Berlin Heidelberg , Berlin Heidelberg , 1996 ) . 19 [ 25 ] J. R. Cash and A. H. Karp , ACM Trans . Math . Software . 16 , 201 ( 1990 ) . [ 26 ] W. H. Press , S. A. Teukolsky , W. T. Vetterling , and B. P. Flannery , Numerical Recipes in Fortran 77 . The Art of Scientific Computing , 2nd ed . ( Cambridge University Press , 1992 ) . [ 27 ] P. J. van der Houwen and B. P. Sommeijer , Z. Angew . Math . Mech . 60 , 479 ( 1980 ) . [ 28 ] J. G. Verwer , W. Hundsdorfer , and B. P. Sommeijer , Numer . Math . 57 , 157 ( 1990 ) . [ 29 ] P. J. van der Houwen , Applied Numerical Mathematics 20 , 261 ( 1996 ) . [ 30 ] J. G. Verwer , Appl . Numer . Math . 22 , 359 ( 1996 ) . [ 31 ] B. P. Sommeijer , L. F. Shampine , and J. G. Verwer , J. Comput . Appl . Math . 88 , 315 ( 1997 ) . [ 32 ] J. G. Verwer , B. P. Sommeijer , and W. Hundsdorfer , J. Comput . Phys . 201 , 61 ( 2004 ) . [ 33 ] D. B. Kirk and W. W. Hwu , Programming Massively Parallel Processors : A Hands-on Approach ( Morgan Kaufmann , Burlington , MA , 2010 ) . [ 34 ] B. Jang , D. Schaa , P. Mistry , and D. Kaeli , IEEE Trans . Parallel Distrib . Syst . 22 , 105 ( 2011 ) . [ 35 ] OpenMP Architecture Review Board , OpenMP application program interface version 3.0 , http://www.openmp.org/mp-documents/spec30.pdf, 2008 . [ 36 ] F. Mazzia and C. Magherini , Test set for initial value problem solvers , release 2.4 , Department of Mathematics , University of Bari and INdAM , Research Unit of Bari , 2008 , Available at http://www.dm.uniba.it/~testset. [ 37 ] N. M. Marinov , Int . J. Chem . Kinet . 31 , 183 ( 1999 ) . [ 38 ] C. K. Law , Combustion Physics ( Cambridge University Press , New York , 2006 ) . [ 39 ] S. Geršgorin , Bulletin de l ' Académie des Sciences de l ' URSS . Classe des sciences mathématiques et na , 749 ( 1931 ) . [ 40 ] R. A. Horn and C. R. Johnson , Matrix Analysis ( Cambridge University Press , 1990 ) . 20