# Efficient Simulation Budget Allocation for Ranking the Topm Designs

Authors: Hui Xiao; Loo Hay Lee
Journal: Discrete Dynamics in Nature and Society
 (2014)

---

## Abstract

We consider the problem of ranking the topm designs out of k alternatives. Using the optimal computing budget allocation framework, we formulate this problem as that of maximizing the probability of correctly ranking the top m designs subject to the constraint of a fixed limited simulation budget. We derive the convergence rate of the false ranking probability based on the large deviation theory. The asymptotically optimal allocation rule is obtained by maximizing this convergence rate function. To implement the simulation budget allocation rule, we suggest a heuristic sequential algorithm. Numerical experiments are conducted to compare the effectiveness of the proposed simulation budget allocation rule. The numerical results indicate that the proposed asymptotically optimal allocation rule performs the best comparing with other allocation rules.


---

## Body

## 1. Introduction

Discrete event system (DES) simulation has been widely used for analyzing and evaluating complex systems since the assumptions for deriving an analytical solution are rarely satisfied in real situation. While DES simulation has been successful in solving many practical problems in a variety of areas such as supply chain systems, healthcare systems, and manufacturing systems, the concerns on the efficiency have never ended [1]. To obtain a statistically significant value, a large number of simulation replications are needed for each design. The performance of each design is then estimated by its sample mean. The ultimate accuracy of this estimator cannot be improved faster than (1/N), where N is the number of simulation replications.Ordinal optimization (OO) which aims to obtain a good estimate through ordinal comparisons although the estimated value is still poor emerges as a way to improve the simulation efficiency [2]. If the goal of the simulation experiment is to identify the good designs instead of finding the accurate estimate of the true performance value, which is true in many real applications, OO can reduce the number of simulation replications significantly [3, 4]. Intuitively, a larger portion of the total simulation replications should be allocated to those designs that are critical in identifying the best design in order to achieve a high probability of correct selection. Based on this idea, an optimal computing budget allocation (OCBA) framework has been proposed to enhance the simulation efficiency further [5, 6]. OCBA focuses on the efficiency of simulation by intelligently allocating further replications based on both the mean and the variance. In parallel with OCBA, two other well-known ranking and selection procedures frequently used in simulation are the indifference zone (IZ) procedure and value information procedure (VIP). The IZ procedure focuses on finding a feasible way to guarantee that the prespecified probability of correct selection can be achieved [7]. The VIP uses the Bayesian posterior distribution to describe the evidence of correct selection and allocates further replications based on maximizing the value information [8, 9]. The three popular procedures of finding the best design are compared in [10]. Extensions of the research in OCBA and ranking and selection include subset selection [11–13], selecting the Pareto set for multiple objective functions [14], selecting the best design subject to stochastic constraints [15, 16], and complete ranking [17]. A detailed summary of the existing work in OCBA can be found in [18].To the best of our knowledge, no previous research has considered the simulation budget allocation for ranking the topm designs out of k alternatives. The developed top m ranking procedure is most useful when the designs for comparison have multiple dimensions of performance measurements with some qualitative criteria such as environmental consideration and political feasibility. Top m ranking procedure can help the decision makers to identify the ranking of the top m designs based on the quantitative performance measurement. Final decision can be made by incorporating other qualitative performance measurements. Hence, the top m ranking gives decision makers a more flexible and people oriented way to support the decision making process. In addition, the top m ranking procedure can also be integrated into some evolutionary search algorithms, where the ranking information of the top solutions is needed in order to determine the search direction of next iteration. For example, the ranking of the top candidate solutions may be needed in each iteration of the genetic algorithm since better candidate solutions are given higher probabilities to reproduce in genetic algorithms.In this paper, we consider the problem of ranking the topm designs out of total k alternatives, where the performance value of each design can only be estimated with noise via simulation. The objective of this paper is to determine how to allocate the simulation replications among the k designs such that the probability of correctly ranking the top m designs can be maximized. The organization of this paper is as follows. Section 2 provides the mathematical formulation of the top m ranking problem using OCBA framework. In Section 3, we derive the convergence rate function of the false ranking probability. Section 4 derives the asymptotically optimal allocation rule using large deviation theory. A sequential allocation algorithm is proposed in Section 5. Section 6 conducts numerical experiments to demonstrate the effectiveness of the proposed simulation budget allocation rule. Finally, we conclude our paper in Section 7.


## 2. Problem Formulation

Consider the problem of ranking the topm designs out of k alternatives. The performance of each design can only be estimated with noise via simulation. The mean performance is used as the ranking criterion. In order to have a steady mean performance value, a large number of simulation replications are needed because of the randomness of individual samples. Given that a total of n simulation replications are available, the objective is to find the best allocation of the total n replications to the k designs in order to maximize the probability of correctly ranking the top m designs.Denote the mean performances of each design byμ1,…,μk such that μ1<⋯<μi<⋯<μk and μi+1-μi≥δ, i=1,…,k-1, where δ is a positive number. Let α=(α1,…,αk) denote the proportion of the total computing budget n to be allocated to each design such that ∑i=1kαi=1, i=1,…,k. Let X-i(αin)=(αin)-1∑j=1αinXij denote the sample mean of design i, where (Xi1,…,Xi,αin) are the samples from population i. We ignore the case when αin is not an integer because we could let αin be ⌊αin⌋, the greatest integer less than αin. The analysis remains unaffected. The objective of this paper is to find the optimal allocation strategy α*=(α1*,…,αk*) such that the probability of correctly ranking the top m designs can be maximized with a fixed limited computing budget n.Given thatμ1<⋯<μi<⋯<μk, the ranking of the top m designs is correctly identified if X-i(αin)≤X-i+1(αi+1n) for all i=1,…,m-1 and if X-m(αmn)≤X-j(αjn) for all j=m+1,…,k. Mathematically, we can write the probability of correctly ranking the top m designs as follows:
(1)P(CRm)=P({⋂i=1m-1(X-i(αin)≤X-i+1(αi+1n))}{⋂j=m+1k}iiiiiii⋂{⋂j=m+1k(X-m(αmn)≤X-j(αjn))}).Hence, an optimal computing budget allocation problem can be formulated by maximizing the probability of correctly ranking the topm designs:
(2)max⁡α1,…,αkP(CRm)s.t.iiiiiiiiiα1+⋯+αi+⋯+αk=1,αi≥0,i=1,…,k.Maximizing the probability of correctly ranking the topm designs is equivalent to minimizing the false ranking probability of the top m designs. Using asymptotical analysis, this is also equivalent to maximizing the convergence rate at which the false ranking probability goes to zero. In this paper, we use large deviation theory to derive this convergence rate function and reformulate the optimization model (2) by maximizing the convergence rate function. The assumptions needed in this paper are stated as follows.Assumption 1.
The performance of every design is independently simulated.
The independence of each design ensures that the samples(Xi1,…,Xi,αin) for each i=1,…,k generated are independent. Thus, the results we obtained will not be affected by the correlations among different designs.
Define the cumulant generating function of sample meanX-i(n) to be Λi(n)(θ)=ln⁡E(eθX-i(n)). The effective domain of any function f:Df→R* is the set {x∈Df:f(x)<∞}, while the range is R*=R∪{+∞}. Let DΛi={θ∈R:Λi(θ)<∞} and Fi={Λi′(θ):θ∈DΛi0}. For any set A, Ao denotes its interior and A- denotes its closure.Assumption 2.
For eachi=1,…,k,(1)
the limitΛi(θ)=lim⁡n→∞⁡(1/n)Λi(n)(nθ) is well defined as an extended real number for all θ;(2)
the origin belongs toDΛi0;(3)
Λi(·) is strictly convex and steep, that is, lim⁡n→∞⁡|Λi′(θn)|=∞, where {θn} is a sequence converging to the boundary point of DΛi;(4)
[μ1,μk]⊂⋂i=1kFi0.Assumption2 implies that μi=Λi′(0) with X-i(n)→μi almost surely when n→∞. Furthermore, it indicates that the sample mean X-i(αin) satisfies the large deviation principle. The last condition of Assumption 2 ensures that the sample mean of every design can take any value between μ1 and μk and P(X-i(αin)≥X-i+1(αi+1n))>0 [19].


## 3. Rate Function of the False Ranking Probability

We now derive the probability of false ranking other than the correct ranking probability, followed by the corresponding derivation of its large deviation principle. Recall that the probability of correctly ranking the topm designs is defined in (1). The probability of falsely ranking the top m designs is simply its complement; that is,
(3)P(FRm)=1-P(CRm)=P{({⋂i=1m-1(X-i(αin)≤X-i+1(αi+1n))}{⋂j=m+1k}){⋂j=m+1k})Ciiiiiiiiiiii⋂{⋂j=m+1k(X-m(αmn)≤X-j(αjn))})C}=P{(⋃i=1m-1(X-i(αin)≥X-i+1(αi+1n))){⋂j=m+1k})iiiiiiiii⋃(⋃j=m+1k(X-m(αmn)≥X-j(αjn)))},
where (·)C represents the complement of (·).P
(
FR
m
) has a lower bound,
(4)lb=max⁡{max⁡1≤i≤m-1P(X-i(αin)≥X-i+1(αi+1n)),(X-m(αmn)≥X-j(αjn))iiiiiiiiiiiiimax⁡j=m+1,…,kP(X-m(αmn)≥X-j(αjn))},
and an upper bound ub=(k-1)×lb such that, assuming the limit exists,
(5)lim⁡n→∞1nln⁡P(FRm)=lim⁡n→∞1nln⁡⁡lb.Theorem3 formally states that the limit exists and the overall convergence rate function is the minimum rate function of each probability. The convergence rate can be understood as the speed at which false ranking probability goes to zero.Theorem 3.
The rate function ofP(FRm) is given by
(6)-lim⁡n→∞1nln⁡P(FRm)=min⁡{min⁡1≤i≤m-1inf⁡x(αiIi(x)+αi+1Ii+1(x)),(αjIj(x)+αmIm(x))iiiiiiiiiiiiiiiimin⁡j=m+1,…,kinf⁡x(αjIj(x)+αmIm(x))},
where I(x)=sup⁡θ∈R⁡(θx-Λ(θ)) is the Fenchel-Legendre transform and Λ(θ)=ln⁡E(eθX).Proof.
If there exists a functionGi(·) such that
(7)lim⁡n→∞1nln⁡P{X-i(αin)≥X-i+1(αi+1n)}=-Gi(αi,αi+1),
for i=1,…,m-1, and
(8)lim⁡n→∞1nln⁡P{X-m(αmn)≥X-j(αjn)}=-Gj(αj,αm),
for j=m+1,…,k.
Then, it can be concluded that(9)lim⁡n→∞1nln⁡P(FRm)=-min⁡{min⁡1≤i≤m-1Gi(αi,αi+1),min⁡j=m+1,…,kGj(αj,αm)}.
Now we are in the position to derive the assumed functionGi(·). Define Yn=(X-i(αin),X-i+1(αi+1n)). The cumulant moment generating function of Yn can be written as
(10)Λn(θi,θi+1)=ln⁡E(eθiX-i(αin)+θi+1X-i+1(αi+1n))=Λi(αin)(θiαin)+Λi+1(αi+1n)(θi+1αi+1n).
Under Assumption2,
(11)lim⁡n→∞1nΛn(nθi,nθi+1)=αiΛi(θiαi)+αi+1Λi+1(θi+1αi+1).
By the Gärtner-Ellis Theorem [20], {Yn,n=1,2,…} satisfies large deviation principle with good rate function which can be expressed as follows:
(12)I(xi,xi+1)=sup⁡θi,θi+1(θixi+θi+1xi+1-αiΛi(θiαi)iiiiiiiiii-αi+1Λi+1(θi+1αi+1))=sup⁡θi(θixi-αiΛi(θiαi))+sup⁡θi+1(θi+1xi+1-αi+1Λi+1(θi+1αi+1))=αisup⁡θi/αi((θiαi)xi-Λi(θiαi))+αi+1sup⁡θi+1/αi+1((θi+1αi+1)xi+1-Λi+1(θi+1αi+1))=αiIi(xi)+αi+1Ii+1(xi+1).
Hence, from large deviation principle,(13)lim⁡n→∞1nln⁡P(X-i(αin)≥X-i+1(αi+1n))=-inf⁡x(αiIi(x)+αi+1Ii+1(x)).
Similarly,(14)lim⁡n→∞1nln⁡P(X-m(αmn)≥X-j(αjn))=-inf⁡x(αjIj(x)+αmIm(x)).
Therefore, the convergence rate function of the false ranking probability can be expressed as follows:(15)-lim⁡n→∞1nln⁡P(FRm)=min⁡{min⁡1≤i≤m-1Gi(αi,αi+1),min⁡j=m+1,…,kGj(αj,αm)}=min⁡{min⁡1≤i≤m-1inf⁡x(αiIi(x)+αi+1Ii+1(x)),min⁡j=m+1,…,kinf⁡x(αjIj(x)+αmIm(x))iiiiiiiiiiiiiiiimin⁡j=m+1,…,kinf⁡x(αjIj(x)+αmIm(x))}.


## 4. Asymptotically Optimal Allocation

The objective is to maximize the probability of correctly ranking the topm designs. This can be achieved by minimizing the false ranking probability. It is also equivalent to maximizing the convergence rate of P(FRm) subject to ∑i=1kαi=1 and αi≥0, for all i=1,…,k. The optimization model in (1) can be reexpressed as
(16)max⁡min⁡{min⁡1≤i≤m-1inf⁡x(αiIi(x)+αi+1Ii+1(x)),(αjIj(x)+αmIm(x))iiiiiiiiiiiiiiiiiiimin⁡j=m+1,…,kinf⁡x(αjIj(x)+αmIm(x))}s.t.iiiiiii∑i=1kαi=1,αi≥0,i∈{1,…,k}.By [19], αiIi(x)+αi+1Ii+1(x) is a strictly increasing concave function. The infimum of concave functions is also concave. Likewise, the minimum of concave functions is a concave function too. Define x(αi,αi+1)=arginf⁡x(αiIi(x)+αi+1Ii+1(x)). As shown in [19], x(αi,αi+1) is the solution to αiIi′(x)  +  αi+1Ii+1′(x)=0 and
(17)∂(αiIi(x(αi,αi+1))+αi+1Ii+1(x(αi,αi+1)))∂αi=Ii(x(αi,αi+1)),∂(αiIi(x(αi,αi+1))+αi+1Ii+1(x(αi,αi+1)))∂αi+1=Ii+1(x(αi,αi+1)).The result can similarly be applied toαjIj(x)+αmIm(x). Therefore, the optimization model (16) is a concave maximization problem and it can be reexpressed as follows:
(18)max⁡zs.t.iiiiiiiαiIi(x(αi,αi+1))+αi+1Ii+1(x(αi,αi+1))≥z,iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii∈{1,…,m-1}iiiiiiiiiiiiαjIj(x(αj,αm))+αmIm(x(αj,αm))≥z,iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiij∈{m+1,…,k}iiiiiiiiiiii∑i=1kαi=1,αi≥0,i∈{1,…,k}.Since model (18) is strictly concave and the functions of α are continuous, a unique optimal solution must exist and the Karush-Kuhn-Tucker (KKT) conditions are necessary and sufficient for global optimality.From the KKT conditions on problem (18), we define a new problem (19) by replacing some inequality signs and forcing αi to be strictly positive:
(19)max⁡zs.t.iiiiiiiα1I1(x(α1,α2))+α2I2(x(α1,α2))=ziiiiiiiiiiiimin⁡{αiIi(x(αi,αi+1))+αi+1Ii+1(x(αi,αi+1)),iiiiiiiiiiiiiiiiiiiiiαi-1Ii-1(x(αi-1,αi))+αiIi(x(αi-1,αi))}=z,iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii∈{2,…,m-1}iiiiiiiiiiiiαjIj(x(αj,αm))+αmIm(x(αj,αm))=z,iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiij∈{m+1,…,k}iiiiiiiiiiii∑i=1kαi=1,αi>0,i∈{1,…,k}.Theorem 4.
Under Assumptions1 and 2 in Section 2, problems (18) and (19) are equivalent; that is, a solution α*=(α1*,…,αk*) is the optimal solution to (18) if and only if it is also an optimal solution to (19).Proof.
We assume that a point satisfying the KKT condition of (18) is also feasible to (19). We first prove the forward and backward assertions. We then prove that the assumption that a point satisfying the KKT condition of (18) is also feasible to (19) is indeed correct.
Supposeα* is the optimal solution to (18). Since the feasible region of (19) is a subset of that of (18), if the optimal solution to (18) is feasible to (19), it must be optimal to (19). Since the KKT conditions are necessary and sufficient for optimality in (18), α*must satisfy the KKT conditions of (18). Hence, α* is feasible to (19). Therefore, if a point satisfies the KKT condition in (18), it must be optimal to (19).
Suppose the optimal solution to (18) is α* and the optimal solution to (19) is α~*, and α*≠α~*. Since the KKT conditions are necessary and sufficient condition to (18), thus, α* must satisfy the KKT conditions. Furthermore, the objective function of (19) is the same as that of (18), and the feasible region of (19) is a subset of that of (18). Therefore, α* must be infeasible to (19). However, we assumed that a point satisfying the KKT conditions of (18) must be feasible to (19). We have thus reached a contradiction. So we must have α*=α~*.
We are now in the position to prove that a point satisfying the KKT conditions of (18) must be feasible to (19). If we let αi=1/k, we can have z>0 for problem (18). However, any αi=0 for i=1,…,m-1 will lead to αi+1inf⁡x⁡Ii+1(x)=αi+1Ii+1(μi+1)=0. If αj=0 for j=m+1,…,k, we will have αminf⁡x⁡Im(x)=αmIm(μm)=0. Therefore, the optimal solution must satisfy αi>0 for every i=1,…,k.
Since the problem (18) is a concave optimization problem, the first order condition is also the optimality condition. According to the KKT conditions, there exist λi≥0, λj≥0, i∈{1,…,m-1}, j∈{m+1,…,k}, and γ>0 such that
(20)∑i=1m-1λi+∑j=m+1kλj=1,(21)λ1I1(x(αi*,αi+1*))=γ,(22)λi-1Ii(x(αi-1*,αi*))+λiIi(x(αi*,αi+1*))=γ,iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii∈{2,…,m-1},(23)λm-1Im(x(αm-1*,αm*))+∑j=m+1kλjIm(x(αj*,αm*))=γ,(24)λjIj(x(αj*,αm*))=γ,j∈{m+1,…,k},(25)λi(αi*Ii(x(αi*,αi+1*))+αi+1*Ii+1(x(αi*,αi+1*))-z)=0,iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii∈{1,…,m-1},(26)λj(αj*Ij(x(αj*,αm*))+αm*Im(x(αj*,αm*))-z)=0,iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiij∈{m+1,…,k}.
Ifλ1=0, γ will be zero. Thus, we could conclude that all λi,λj=0, j=m+1,…,k, by putting γ=0 into (21) to (24). Substituting λ1=0 and γ=0 into (22) and (23) will result in λi=0, i=2,…,m-1. However, this contradicts with (20) which requires at least one λi,λj>0. Thus, we conclude that λ1>0 and γ>0 because Ii(x(αi*,αi+1*)), i=1,…,k-1, is strictly positive. Similarly, we could conclude that λj>0, j=m+1,…,k, from (24), and max⁡{λi,λi+1}>0, i=1,…,m-1, from (22).
Based on the results thatλ1>0, λj>0, j∈{m+1,…,k}, max⁡{λi,λi+1}>0, i=2,…,m-1, and constraints of (18), we have the following equality from the complementary slackness condition in (25) and (26). For i∈{2,…,m-1}, j∈{m+1,…,k},
(27)z=α1*I1(x(α1*,α2*))+α2*I2(x(α1*,α2*))=αj*Ij(x(αj*,αm*))+αm*Im(x(αj*,αm*))=min⁡{αi*Ii(x(αi*,αi+1*))+αi+1*Ii+1(x(αi*,αi+1*)),iiiiiiiiiαi-1*Ii-1(x(αi-1*,αi*))+αi*Ii(x(αi-1*,αi*))}.
So we have proved the assertion that a point satisfying the KKT conditions of (18) must be feasible to (19). This completes the proof of Theorem 4.Therefore, we could conclude that the optimal allocation rule to rank the topm designs α*=(α1*,…,αk*) solves
(28)∑i=1kαi=1,αi>0,min⁡{αi*Ii(x(αi*,αi+1*))+αi+1*Ii+1(x(αi*,αi+1*)),iiiiiiiiiαi-1*Ii-1(x(αi-1*,αi*))+αi*Ii(x(αi-1*,αi*))}=α1*I1(x(α1*,α2*))+α2*I2(x(α1*,α2*))=αj*Ij(x(αj*,αm*))+αm*Im(x(αj*,αm*)),iiiiiiiiiiiiii=2,…,m-1;j=m+1,…,k.Suppose that the performance of each design follows the normal distribution; that is,Xi~N(μi,σi2), i=1,…,k. Equation (28) can be rewritten as follows:
(29)∑i=1kαi=1,αi>0,min⁡{(μi-μi+1)22(σi2/αi*+σi+12/αi+1*),(μi-μi-1)22(σi2/αi*+σi-12/αi-1*)}=(μ1-μ2)22(σ12/α1*+σ22/α2*)=(μj-μm)22(σj2/αj*+σm2/αm*),iiiiiiiiiiiiiiiiiiii=2,…,m-1;j=m+1,…,k.


## 5. Sequential Allocation

Based on the results from Theorem4, we can compute the simulation budget allocation rule using (28) for design performances with any arbitrary distributions or using (29) for normally distributed performances if the parameters of the design performance distribution are given. However, no information on the design performance distribution is known before simulation experiments are conducted. To overcome this dilemma, we suggest a heuristic sequential allocation algorithm in order to implement the allocation rule. The algorithm for sequential allocation is summarized in Algorithm 1.Algorithm 1:Sequential allocation algorithm.
INPUT:m,k,n,n0,Δ (n-kn0 is a multiple of Δ and n0≥5)
INITIALIZE: Performn0 simulation replications for all designs.
l←0;N1l=⋯=Nil=⋯=Nkl
WHILE∑i=1kNil<n  DO
(1) Increase the computing budget by Δ
(2) Use (28) to determine the new budget allocation rule, that is, Nil+1,i=1,…,k.
The population distribution parameters are estimated by the sample statistics.
(3) Simulate additionalmax⁡⁡(0,Nil+1-Nil) for each design i=1,…,k.
END OF WHILEDefinel to be the iteration number and define Nil, i=1,…,k to be the total number of simulation replications that have been allocated to design i up to iteration l. n is the total number of simulation replications available. Δ is the number of incremental simulation replications for each iteration. n0 is the initial number of simulation runs for each design.As the simulation continues, designi will be ranked number i for all i≤m. The ranking of the top m designs may change from iteration to iteration although it will converge to the true ranking when the total computating budget goes to infinity. When the ranking of the top m designs changes, the budget allocation in the loop will be applied immediately. Therefore, the actual proportion of budget for every system will converge to the optimal proportion when the number of iterations is sufficiently large.Furthermore, we need to take note ofn0, the initial number of replications for every design. n0 cannot be too small because the estimation of the rate function can be poor especially when the variance of the performance is large. On the other hand, if n0 is too large, some designs will be allocated excessively compared with their optimal allocation numbers. When the total budget is very limited, designs that need more replications may suffer from large n0 and this would eventually affect the simulation results. Other than the initial number of replications, the incremental budget Δ is also important in the implementation process. Large Δ results in wasting of budget, while small Δ will lead to expensive computation in the loop.It is also worthy to note that there will be significant variation in the estimate of the performance value. Denote the empirical cumulant generating function and rate function of systemi as Λi(n)(θ)=ln⁡((1/n)∑j=1neθX-ij) and Ii(n)(x)=sup⁡θ⁡{θx-Λi(n)(θ)}, respectively. In Algorithm 1, we are using Gi(n)(αi,αi+1) to estimate Gi(αi,αi+1) for the optimal allocation rule. Thus, the true optimal allocation αi* is actually estimated by αi*(n). As argued in [19], the empirical estimation of the optimal allocation is consistent; that is, αi*(n)→αi*, for all i=1,…,k almost surely when n→+∞.


## 6. Numerical Experiments

In this section, we test the proposed simulation budget allocation rule for ranking the topm designs by comparing it with different allocation rules: equal allocation which simulates each design equally and the OCBA-m procedure [12]. Although OCBA-m only considers the selection of the top m designs and do not aim to identify the ranking of the top m designs, we can use it here for benchmarking purpose. In all the experiments below, the performance of each design is assumed to follow the normal distribution. Therefore, the optimal allocation rule can be obtained by solving (29). The assumption of normal distribution is generally held in simulation experiments since the output is obtained from an average performance or batch means, so that the central limit theorem effect holds.


### 6.1. Computing Budget Allocation Rules

Equal Allocation. The simulation replications are allocated equally to each design; that is, αi=1/k, i=1,…,k. This is the simplest allocation rule and it can serve as a benchmark for other allocation procedures.Top
m
Ranking (OCBA-Rm). The simulation budget allocation is derived using (29) and implemented using the sequential allocation algorithm proposed in Section 5. The allocation rule is solved by using the solver “fminimax” in Matlab.OCBA-m [12]. Sequential allocation algorithm is used to implement this allocation rule. The simulation budget allocation rule is determined by the following equation:
(30)α1l+1s12/δ12=α2l+1s22/δ22=⋯=αkl+1sk2/δk2,
where si2 is the sample variance for design i, δi2=(X-i-c)2, and c=(sm+1X-m+smX-m+1)/(sm+1+sm) as defined in [12].


### 6.2. Numerical Results for Different Allocation Procedures

To compare the performance of the procedures, we carried out numerical experiments for the different allocation procedures discussed above. In comparing the procedures, the effectiveness of the procedures is measured by the probability of correctly ranking the topm designs (P(CRm)) which is estimated by the fraction of the times that the procedure successfully identifies the correct ranking of the top m designs out of 10,000 independent simulation runs.Each of the procedures simulates each of thek designs for n0=20 replications initially as recommended in [1, 12]. The simulation budget is increased by Δ=40 for each iteration. Table 1 summarizes the mean and variance for the three experiments that will be conducted in this section. In this experiment, the total number of designs is 20, that is, k=20 and the objective is to identify the ranking of the top 5 designs, that is, m=5.Table 1
Parameters for the numerical experiments.
Design
Equal spacing
Equal variance
Increasing spacing decreasing variance
Mean
Variance
Mean
Variance
Mean
Variance
I
2
400
1
100
1
400
II
4
361
2
100
2
361
III
6
324
4
100
4
324
IV
8
289
7
100
7
289
V
10
256
11
100
11
256
VI
12
225
16
100
16
225
VII
14
196
22
100
22
196
VIII
16
169
29
100
29
169
IX
18
144
37
100
37
144
X
20
121
46
100
46
121
XI
22
100
56
100
56
100
XII
24
81
67
100
67
81
XIII
26
64
79
100
79
64
XIV
28
49
92
100
92
49
XV
30
36
106
100
106
36
XVI
32
25
121
100
121
25
XVII
34
16
137
100
137
16
XVIII
36
9
154
100
154
9
XIX
38
4
172
100
172
4
XX
40
1
191
100
191
1Figure1 shows the numerical comparison for the three allocation rules, where (a) is for the equal spacing scenario, (b) is for the equal variance scenario, and (c) is for the increasing spacing but decreasing variance scenario.Probability of correctly ranking the topm designs under three different allocation rules.
(a)
(b)
(c)The experiment results show that the proposed simulation budget allocation rule OCBA-Rm performs the best in all three experiments. It is also interesting to note that OCBA-m, which performs significantly better than EA when the objective is selecting the top m designs, fares much worse than EA in all three experiments when the objective is to identify the ranking of the top m designs. This is because OCBA-m only focuses on distinguishing the set of the top m designs and the set of the nontop m designs without considering the ranking within the top m designs. Based on the numerical results, we can see that it is important to derive the OCBA-Rm when the ranking of the top m alternatives is needed. By using the proposed OCBA-Rm allocation rule, significant number of simulation budgets can be saved compared with EA and OCBA-m.


## 6.1. Computing Budget Allocation Rules

Equal Allocation. The simulation replications are allocated equally to each design; that is, αi=1/k, i=1,…,k. This is the simplest allocation rule and it can serve as a benchmark for other allocation procedures.Top
m
Ranking (OCBA-Rm). The simulation budget allocation is derived using (29) and implemented using the sequential allocation algorithm proposed in Section 5. The allocation rule is solved by using the solver “fminimax” in Matlab.OCBA-m [12]. Sequential allocation algorithm is used to implement this allocation rule. The simulation budget allocation rule is determined by the following equation:
(30)α1l+1s12/δ12=α2l+1s22/δ22=⋯=αkl+1sk2/δk2,
where si2 is the sample variance for design i, δi2=(X-i-c)2, and c=(sm+1X-m+smX-m+1)/(sm+1+sm) as defined in [12].


## 6.2. Numerical Results for Different Allocation Procedures

To compare the performance of the procedures, we carried out numerical experiments for the different allocation procedures discussed above. In comparing the procedures, the effectiveness of the procedures is measured by the probability of correctly ranking the topm designs (P(CRm)) which is estimated by the fraction of the times that the procedure successfully identifies the correct ranking of the top m designs out of 10,000 independent simulation runs.Each of the procedures simulates each of thek designs for n0=20 replications initially as recommended in [1, 12]. The simulation budget is increased by Δ=40 for each iteration. Table 1 summarizes the mean and variance for the three experiments that will be conducted in this section. In this experiment, the total number of designs is 20, that is, k=20 and the objective is to identify the ranking of the top 5 designs, that is, m=5.Table 1
Parameters for the numerical experiments.
Design
Equal spacing
Equal variance
Increasing spacing decreasing variance
Mean
Variance
Mean
Variance
Mean
Variance
I
2
400
1
100
1
400
II
4
361
2
100
2
361
III
6
324
4
100
4
324
IV
8
289
7
100
7
289
V
10
256
11
100
11
256
VI
12
225
16
100
16
225
VII
14
196
22
100
22
196
VIII
16
169
29
100
29
169
IX
18
144
37
100
37
144
X
20
121
46
100
46
121
XI
22
100
56
100
56
100
XII
24
81
67
100
67
81
XIII
26
64
79
100
79
64
XIV
28
49
92
100
92
49
XV
30
36
106
100
106
36
XVI
32
25
121
100
121
25
XVII
34
16
137
100
137
16
XVIII
36
9
154
100
154
9
XIX
38
4
172
100
172
4
XX
40
1
191
100
191
1Figure1 shows the numerical comparison for the three allocation rules, where (a) is for the equal spacing scenario, (b) is for the equal variance scenario, and (c) is for the increasing spacing but decreasing variance scenario.Probability of correctly ranking the topm designs under three different allocation rules.
(a)
(b)
(c)The experiment results show that the proposed simulation budget allocation rule OCBA-Rm performs the best in all three experiments. It is also interesting to note that OCBA-m, which performs significantly better than EA when the objective is selecting the top m designs, fares much worse than EA in all three experiments when the objective is to identify the ranking of the top m designs. This is because OCBA-m only focuses on distinguishing the set of the top m designs and the set of the nontop m designs without considering the ranking within the top m designs. Based on the numerical results, we can see that it is important to derive the OCBA-Rm when the ranking of the top m alternatives is needed. By using the proposed OCBA-Rm allocation rule, significant number of simulation budgets can be saved compared with EA and OCBA-m.


## 7. Conclusion

In this paper, we study the problem of simulation budget allocation of ranking the topm designs out of k alternatives. Based on the large deviation theory, we have derived the asymptotically optimal allocation rule to maximize the probability of correctly ranking the top m designs. In addition, a heuristic sequential allocation algorithm is suggested to implement the simulation budget allocation rule. Numerical experiments are conducted to compare the effectiveness of the proposed simulation budget allocation with some existing allocation procedures. The contribution of this paper is twofold. From a ranking and selection perspective, we offer a heuristic for ranking the top m designs out of k alternatives, where our empirical studies show that it can be more efficient than the existing methods. From the computing budget allocation perspective, our heuristic illustrates how the previous OCBA method for identifying the single best design can be modified to identify the ranking of the top m designs. In particular, we derive the asymptotically optimal allocation using large deviation theory, which allows us to remove the assumption that the performance of each design must be normally distributed.



---

*Source: 195054-2014-06-18.xml*