Proceedings of the 2020 Winter Simulation Conference
K.-H. Bae, B. Feng, S. Kim, S. Lazarova-Molnar, Z. Zheng, T. Roeder, and R. Thiesing, eds.

CONTEXT-DEPENDENT RANKING AND SELECTION UNDER A BAYESIAN FRAMEWORK
Haidong Li

Henry Lam

Department of Industrial Engineering andManagement
Peking University
5 Yiheyuan Road
Beijing 100871, P. R. CHINA

Department of Industrial Engineering andOperations Research
Columbia University
500 W. 120th Street
New York, NY 10027 USA

Zhe Liang

Yijie Peng

Department of Management Science andEngineering
Tongji University
1239 Siping Road
Shanghai 200092, P. R. CHINA

Department of Management Science andInformation Systems
Peking University
5 Yiheyuan Road
Beijing 100871, P. R. CHINA

ABSTRACT
We consider a context-dependent ranking and selection problem. The best design is not universal but
depends on the contexts. Under a Bayesian framework, we develop a dynamic sampling scheme forcontext-dependent optimization (DSCO) to efficiently learn and select the best designs in all contexts. The
proposed sampling scheme is proved to be consistent. Numerical experiments show that the proposed
sampling scheme significantly improves the efficiency in context-dependent ranking and selection.
1

INTRODUCTION

Simulation is a powerful tool for optimizing complex stochastic systems. We consider a simulation
optimization problem of selecting the best design under different contexts. The mean performances of each
design under each context are unknown and can only be estimated via simulation. The performance of each
design depends on the contexts, and thus the best design is also context-dependent. For example, in patientspecific treatment regimen-making (Kim et al. 2011), therapies and patients can be regarded as designs andcontexts, respectively. We aim to determine the most effective medical treatment for each patient. Other
examples include movie recommendation (Liu et al. 2009) and automated asset management (Faloon andScherer 2017).
For any fixed context, we aim to find the best design among a finite set of alternatives, which is referred
to as ranking and selection (R&S) in the literature. R&S procedures intelligently allocate simulation
replications to efficiently learn the best design. The probability of correct selection (PCS) is used as ameasure to evaluate the efficiency of sampling procedure in R&S. In our problem, the best design is
not universal but context-dependent. In this work, we develop a sampling scheme to allocate simulation
replications for efficiently learning the best design in each context, and the worst-case probability of correct
selection (PCSW ) under all contexts is used to measure the efficiency of our sampling scheme.

Li, Lam, Liang, and Peng
There are the frequentist and Bayesian branches in R&S. See Kim and Nelson (2006) and Chenet al. (2015) for overviews. Frequentist procedures (e.g., Rinott 1978, Kim and Nelson 2001, Luo et al.
2015) allocate simulation replications to guarantee a pre-specified PCS level, whereas Bayesian procedures(e.g. Chen et al. 2000, Chick and Frazier 2012, Gao et al. 2017) aim to either maximize the PCS or
minimize the expected opportunity cost subject to a given simulation budget. Bayesian procedures usuallyachieve better performance than frequentist procedures under a given simulation budget, but they typicallydo not provide a guaranteed PCS. Peng et al. (2018) propose a stochastic control problem to formulate
the sequential decision for the Bayesian framework of R&S. Using a value function approximation, they
develop efficient sampling procedures to approximate the optimal policy for maximizing the posterior PCS.
The literature on context-dependent R&S is sparse relative to the actively studied R&S problem insimulation. Contexts are also known as the covariates, side information, or auxiliary quantities. To thebest of our knowledge, the study of Shen et al. (2017) is the first research for this problem. They assume
a linear relationship between the response of a design and the contexts, and develop sampling proceduresto provide a guarantee on PCS for all contexts. Li et al. (2018) further extend the result in Shen et al.
(2017) to high-dimensional contexts and general dependence between the mean performance of a design
and the contexts. The aforementioned two studies adopt the Indifference Zone paradigm in the frequentist
branch. Gao et al. (2019) adopt an optimal computing budget allocation (OCBA) approach in R&S, andsolve the problem by identifying the rate-optimal budget allocation rule.
In our work, we consider the context-dependent R&S problem under a stochastic control framework,and formulate the sequential sampling decision as a stochastic dynamic programming problem. Under aBayesian framework, we provide an efficient scheme to update the posterior information for each designcontext pair and a dynamic sampling policy based on the sequentially updated posterior information toefficiently learn the performance of each design. The proposed sampling scheme is proved to be consistentand is demonstrated to perform well empirically.
The rest of the paper is organized as follows. In Section 2, we formulate the context-dependent R&S
problem. Section 3 proposes an efficient dynamic sampling scheme to approximately solve the problem.
Section 4 presents numerical results, and the last section concludes the paper and outlines future directions.
2

PROBLEM FORMULATION

Suppose there are n different designs. For i = 1, . . . , n, the performance yi (x) of design i depends ona vector of context x = (x1 , . . . , xd )> for x ∈ X ⊆ Rd . Each dimension of x could be a continuous
variable, discrete variable, or categorical variable. In this study, we assume that X contains a finite numberof m possible contexts x1 , . . . , xm , and all the contexts are known upfront. Our objective is to correctly
select the best design for a given value of x (see Figure 1 for an illustration), i.e., identify arg maxi yi (x).
The offline optimization results can be used for online decision making. For example, in personalized
medicine (Bertsimas et al. 2017), doctors use simulation to determine the best treatment regimen (design) for
patients (context) with different biometric characteristics; in personalized movie recommendation (Zhang
et al. 2016), we aim to recommend the most favorite movie (design) for the corresponding user (context).
The performances yi (xj ) are unknown and can only be learned via sampling. We assume that foreach design and context, the simulation observations are i.i.d. normally distributed, i.e., Yi,t (xj ) ∼N (yi (xj ), σi2 (xj )), i = 1, . . . , n, j = 1, . . . , m, t ∈ Z+ , and the replications are independent acrossdifferent designs or different contexts. The variance σi2 (xj ) in the sampling distribution is assumed to be
known in this study and the sample estimate is used as a plug-in for the true value in practice. Suppose theprior distribution of yi (xj ) is N (µ0 , σ02 ). By conjugacy (Gelman et al. 2014), the posterior distribution ofyi (xj ) is a normal distribution with the posterior mean:
#
" Pt
ij
µ0
(t)
2 (t)
h=1 Yi,h (xj )+ 2 ,µij = (σij )
σi2 (xj )σ0Li, Lam, Liang, and Pengand the posterior variance:
.

2 (t)
(σij
) =1


tij
1
,
+
σi2 (xj ) σ02

P P
where tij is the number of samples allocated to estimate yi (xj ) after allocating t = ni=1 m
j tij samples.
As for a normal distribution with unknown variance, the posterior distribution of yi (xj ) has a normal-gamma
conjugate prior (DeGroot 2005), and the corresponding analysis in context-dependent R&S problem is left
for future research.

Figure 1: Selecting the best (red) design under each context.
Since sampling could be expensive, the total number of samples is usually limited. Moreover, wheneither n or m is relatively large, it would be practically infeasible to estimate all performances accurately
for each design i and each context xj . Given the information of s allocated samples, the selection is topick the designs with the largest posterior estimates in each context. Under a fixed context xj , the qualityof the selection for the best design is measured by the probability of correct selection (PCS),
PCS(xj ) = P yh1ijs (xj ) > yhiijs (xj ), i 6= 1 Es ,where P (·|Es ) denotes the posterior probability, Es is the information set of all s samples, and hiijs , i =1, . . . , n, are the ranking indices for context xj such that
(s)

(s)

µh1ijs j > · · · > µhnijs j .
In this study, we aim to provide the best design for all the x that might possibly appear, and therefore
need a measure for evaluating the quality of the selection over the entire context space X . Specifically, we
adopt the worst-case probability of correct selection over X :PCSW = min PCS(x).
x∈X

This measure has been used in contextual R&S (Gao et al. 2019), and is similar to the worst-case performancein robust optimization (Bertsimas et al. 2011) and R&S with input uncertainty (Gao et al. 2017, Fan et al.
2020).

Li, Lam, Liang, and Peng
We aim to provide a dynamic sampling scheme As to maximize the PCSW :max min P yh1ijs (xj ) > yhiijs (xj ), i 6= 1 Es .

(1)

As j=1,...,mThe dynamic sampling scheme As is a sequence of maps As (·) = (A1 (·), . . . , As (·)). Based on sampling
observations Et−1 , At (Et−1 ) ∈ {(i, j) : 1 ≤ i ≤ n, 1 ≤ j ≤ m} allocates the t-th sample to estimate theperformance of design i in context xj . Similar to that in Peng et al. (2016) and Peng et al. (2018), thesequential sampling decision such as (1) can be formulated as a stochastic control (dynamic programming)
problem. The expected payoff for a sampling scheme As can be defined recursively byVs (Es ; As ) , min P yh1ijs (xj ) > yhiijs (xj ), i 6= 1|Es ,j=1,...,mand for 0 ≤ t < s,hiVt (Et ; As ) , E Vt+1 (Et ∪ {Yi,tij +1 (xj )}; As ) Et(i,j)=At+1 (Et ).

Then, the optimal sampling scheme is well defined byA∗s , arg max V0 (θ0 ; As ),As

where θ0 is parameter in prior distribution. It is important to note that the definition of decision variable inour study is different from the one in R&S. For the R&S problem, the decision is to choose an alternativei in sampling, whereas our decision is to choose a design-context pair (i, j) in sampling.
3

DYNAMIC SAMPLING SCHEME

In principle, backward induction can be used to solve the stochastic dynamic programming problem, but
it suffers from curse-of-dimensionality (Peng et al. 2018). To derive a dynamic sampling scheme withan analytical form, we adopt approximate dynamic programming (ADP) schemes which make dynamic
decision based on a value function approximation (VFA) and keep learning the VFA with decisions moving
forward.
By treating any t-th step as the last step, the value function in our problem is
Vt (Et ) = min P yh1ijt (xj ) > yhiijt (xj ), i 6= 1 Et .
(2)
j=1,...,m(t)

2 )(t) , i = 1, . . . , n,Conditioned on Et , yi (xj ) follows a normal distribution with mean µij and variance (σijj = 1, . . . , m. Therefore, the joint distribution of vector (yh1ijt (xj )−yh2ijt (xj ), . . . , yh1ijt (xj )−yhnijt (xj ))
(t)

(t)

(t)

(t)

follows a joint normal distribution with mean vector (µh1ijt j − µh2ijt j , . . . , µh1ijt j − µhnijt j ) and covariance22matrix Γ0 ΛΓ, where Λ , diag((σh1i)(t) , . . . , (σhni)(t) ), and
jt j
jt j



1
1
−1 0


Γ ,  0 −1
 ..
..
 .
.
0
0

1
0
0
..
.

···
···
···

···
0 ···


1
0

0

.. 
. 
−1 n×(n−1),

Li, Lam, Liang, and Peng
The posterior probability
P yh1ijt (xj ) > yhiijt (xj ), i 6= 1 Et

(3)is an integral of the multivariate standard normal density over a region encompassed by some hyperplanes.
Under a given context, the above probability of correct selection is consistent with that in R&S literature,
which allows us to use a similar approximation developed in Peng et al. (2018). As shown in Figure 2,
the integral over a maximum tangent inner ball in the shadowed region can capture the main body of theintegral over entire region due to the exponential decay of the normal density. It can be shown that the
integral over the omitted domain decreases to zero exponentially as t goes to infinity. Therefore, we use
the volume of the ball as an approximation for the PCS in (3). Specifically, we use a VFA for the valuefunction (2) given by(t)

Vet (Et ) =

min

min

(t)

(µh1ijt j − µhiijt j )2(t) + (σ 2(t)j=1,...,m i6=1 (σ 2h1ijt j )hiijt j ).

Figure 2: Approximation of PCS in (3).
At any step t, we treat the (t + 1)-th step as the last step and try to maximize the expected VFA by
allocating the (t + 1)-th sample to a design-context pair (r, q):hiVet (Et ; (r, q)) , E Vet+1 (Et ∪ {Yr,trq +1 (xq )}) Et .
Further, we apply a certainty equivalent approximation (Bertsekas 1995) to the VFA looking one-stepahead:ihih
E Vet+1 (Et ∪ {Yr,trq +1 (xq )}) Et ≈ Vet+1 Et ∪ E Yr,trq +1 (xq ) Et .

Li, Lam, Liang, and Peng
Then the VFA looking one-step ahead can be calculated, i.e.,(t)

Vbt (Et ; (r, q)) ,

min

min

(t)

(µh1ijt j − µhiijt j )2(t;(r,q)) + (σ 2(t;(r,q))j=1,...,m i6=1 (σ 2h1ijt j )hiijt j )

,

(4)

where

 .
tij + 11
 1
, (i, j) = (r, q),+2 (t;(r,q))(σij)=
σi2 (xj ) σ022 )(t) , otherwise.
(σij
We propose the following dynamic sampling scheme for context-dependent optimization (DSCO):)(
At+1 (Et ) = (r∗ , q ∗ ) Vbt (Et ; (r∗ , q ∗ )) = max Vbt (Et ; (r, q)) ,
(r,q)

(5)

which maximizes VFA looking one-step ahead.
DSCO uses the information on the posterior
performances of means and variances of context-dependent

(t)
(t)
2
2
(t;(r,q))2(t;(r,q))design. We note that (µh1ijt j − µhiijt j ) and (σh1ijt j )+ (σhiijt j )are the squared mean andvariance of the posterior distribution of the difference in performances of design, respectively. Therefore,
equation (4) can be rewritten as
min min 1/c2v (i, j),j=1,...,m i6=1

where cv (i, j) is the coefficient of variation (CV, or sometimes called noise-signal ratio) of the posterior(yh1ijt (xj ) − yhiijt (xj )). DSCO minimizes the maximum of cv (i, j), which is intuitively reasonable sincelarge cv (i, j) implies high difficulty in comparing yh1ijt (xj ) and yhiijt (xj ) from the posterior information.
DSCO sequentially allocates each sample to estimate performance of design to reduce the CV, focusing
on the worst-case context and the pair most difficult to rank. Under the Bayesian framework, consistency
means that for any possible realization of the true means for different alternatives from the prior distribution,
the selected best alternative based on the posterior means of different alternatives would eventually be thetrue best alternative for almost every sample path in the sampling probability space as the simulation budget
goes to infinity (Frazier and Powell 2011). DSCO is proved to be consistent in the following theorem.
Theorem 1 The proposed DSCO is consistent, i.e., ∀j = 1, . . . , m,lim h1ijt = h1ij a.s.

t→+∞where hiij , i = 1, . . . , n, are the ranking indices for context xj such that yh1ij (xj ) > · · · > yhnij (xj ) andthus h1ij denotes the true best design in context xj .
Proof.
We only need to prove that each yi (xj ) will be sampled infinitely often a.s. following DSCO
scheme, and the consistency will follow by the law of large numbers. Suppose yi (xj ) is only sampled
finitely often and yr (xq ) is sampled infinitely often. Therefore, there exists a finite number N0 such thatyi (xj ) will stop receiving replications after the sampling number t exceeds N0 . Thus we have2 (t)lim (σij) > 0,2 (t)lim (σrq) = 0.

t→+∞

t→+∞

lim

h
i
2 (t)
2 (t;(r,q))(σrq) − (σrq)= 0,

By noticing thatt→+∞

Li, Lam, Liang, and Peng
lim

t→+∞

we havehi
2 (t)2 (t;(i,j))(σij) − (σij
)
> 0,

i
h
Vbt (Et ; (r, q)) − Vet (Et ) = 0 a.s.,t→+∞hilim Vbt (Et ; (i, j)) − Vet (Et ) > 0 a.s.,
lim

t→+∞

which contradicts with the sampling rule (5) that the design-context pair with the largest Vbt (Et ; (i, j)) is
sampled. Therefore, the proposed DSCO scheme must be consistent.
One can also prove the proposed DSCO achieves the following asymptotically optimal sampling ratio
obtained in Gao et al. (2019) as the number of samples goes to infinity:
t2h1i j
j

2
σh1i
(xj )
j

(yh1ij (xj ) − yhiij (xj ))222σh1i(xj )/th1ij j + σhii
(xj )/thiij j
j

j

=

=

n
X

t2hii jj

σ 2 (xj )i=2 hii, j = 1, . . . , m,j(yh1ij 0 (xj 0 ) − yhi0 ij 0 (xj 0 ))222
σh1i(xj 0 )/th1ij 0 j 0 + σhi
0 i 0 (xj 0 )/thi0 ij 0 j 0
0j, i, i0 = 2, . . . , n, j, j 0 = 1, . . . , m.

j

The proof is rather technical and will be included in later work.
4

NUMERICAL RESULTS

In this section, we conduct numerical experiments to test the performance of different sampling proceduresfor context-dependent R&S problems. The proposed DSCO is compared with the equal allocation (EA),
two-stage indifference-zone (IZ) procedure in Shen et al. (2017), and the contextual optimal computing
budget allocation (C-OCBA) in Gao et al. (2019):•••

EA: This procedure allocates the same number of samples to any design-context pair.
IZ: At first stage, this procedure takes n0 independent samples of each design-context pair and calcu2 ; At second stage, take max{dh2 S 2 /δ 2 e − n , 0} additional independent
lates sample variances Sij
0ij
samples of design i in context xj , where h and δ are IZ parameters.
C-OCBA: This procedure sequentially allocates each sample to achieve the asymptotically optimal
sampling ratio.

In all numerical examples, the statistical efficiency of the sampling procedures is measured by the PCSW
estimated by 10,000 independent experiments. The PCSW is reported as a function of the sampling budget
in each experiment.
Example 1: 10 × 10 design-context pairs
We test our proposed DSCO in a synthetic case with 10 designs and 10 contexts. The performances ofeach design for each context are generated as follows:yi (xj ) ∼ N (50, 32 ), i = 1, . . . , 10, j = 1, . . . , 10,
which means there is no clear performance clustering structure and all design-context pairs belong to acommon cluster. For design i and context xj , samples are drawn independently from a normal distribution
N (yi (xj ), σi2 (xj )), where σi (xj ) ∼ U (8, 12). We set the number of initial replications as n0 = 5 for each
design-context pair.
In Figure 3, we can see that DSCO and C-OCBA perform better than IZ and EA, which could be
attributed to the reason that EA utilizes no sample information and IZ only utilizes sample variances while
the other two sampling procedures utilize the information in the posterior means and variances. In order

Li, Lam, Liang, and Peng

Figure 3: PCSW of the four sampling procedures in Example 1.
to attain PCSW = 80%, the number of samples consumed by DSCO is 2000, while EA, IZ and C-OCBA
require more than 2800 samples. That is to say DSCO reduces the sampling budget by more than 28%.
Compared with C-OCBA, the performance enhancement of DSCO could be attributed to that it has atheoretical support for the finite-sample performance in its derivation via VFA.
Example 2: 30 × 30 design-context pairs
In this example, our proposed DSCO is tested in a larger synthetic case with 30 designs and 30 contexts.
The performances of each design for each context are generated as follows:yi (xj ) ∼ N (50, 152 ), i = 1, . . . , 30, j = 1, . . . , 30.
For design i and context xj , samples are drawn independently from a normal distribution N (yi (xj ), σi2 (xj )),where σi (xj ) ∼ U (4, 6). We set the number of initial replications as n0 = 5 for each design-context pair.
Figure 4 illustrates the performance of the four sampling procedures. In the presence of more alternative
designs and contexts, larger sampling budget is needed to reach the same PCSW level. Similar to Example
1, DSCO remains as the most efficient sampling procedure among the four, and C-OCBA is better than
IZ and EA. Comparing the result in this example with that in Example 1, we can see that the advantage
of DSCO is more significant when the numbers of designs and contexts become larger. In order to attain
PCSW = 90%, DSCO consumes less than 30,000 samples, while EA, IZ and C-OCBA require more than
45,000 samples. That is to say DSCO reduces the sampling budget by more than 33%. Moreover, the gap
between the PCSW of DSCO and those of EA, IZ, and C-OCBA widens as the sampling budget increases
in the experiment.
5

CONCLUSION

This paper studies a sample allocation problem for context-dependent R&S. We propose an efficient sampling
procedure named DSCO, which maximizes the worst-case probability of correct selection over the entire
context space. Numerical experiments demonstrate that DSCO is significantly more efficient than the other
tested sampling procedures. Future research includes the asymptotic analysis for the sampling ratio of theLi, Lam, Liang, and Peng

Figure 4: PCSW of the four sampling procedures in Example 2.
proposed sequential sampling procedure. Considering the performance clustering between different designs
and contexts could also be an interesting future work.
ACKNOWLEDGMENTS
This work was supported in part by the National Science Foundation of China (NSFC) under Grants
71901003, 91846301, 71790615, 71720107003, 71690232, and 61603321, by the National Science Foundation under Awards ECCS-1462409, CMMI-1462787, CAREER CMMI-1834710 and IIS-1849280, andby the scholarship from China Scholarship Council (CSC) under the Grant CSC.
REFERENCES
Bertsekas, D. P. 1995. Dynamic programming and optimal control, Volume 1. Athena scientific Belmont,
MA.
Bertsimas, D., D. B. Brown, and C. Caramanis. 2011. “Theory and applications of robust optimization”.
SIAM review 53(3):464–501.
Bertsimas, D., N. Kallus, A. M. Weinstein, and Y. D. Zhuo. 2017. “Personalized diabetes management
using electronic medical records”. Diabetes care 40(2):210–217.
Chen, C.-H., S. E. Chick, L. H. Lee, and N. A. Pujowidianto. 2015. “Ranking and selection: efficientsimulation budget allocation”. In Handbook of Simulation Optimization, 45–80. Springer.
Chen, C.-H., J. Lin, E. Yücesan, and S. E. Chick. 2000. “Simulation budget allocation for further enhancing
the efficiency of ordinal optimization”. Discrete Event Dynamic Systems 10(3):251–270.
Chick, S. E., and P. Frazier. 2012. “Sequential sampling with economics of selection procedures”. Management Science 58(3):550–569.
DeGroot, M. H. 2005. Optimal statistical decisions, Volume 82. John Wiley & Sons.
Faloon, M., and B. Scherer. 2017. “Individualization of robo-advice”. The Journal of Wealth Management 20(1):30–36.
Fan, W., L. J. Hong, and X. Zhang. 2020. “Distributionally robust selection of the best”. Management
Science 66(1):190–208.

Li, Lam, Liang, and Peng
Frazier, P. I., and W. B. Powell. 2011. “Consistency of sequential Bayesian sampling policies”. SIAM
Journal on Control and Optimization 49(2):712–731.
Gao, S., W. Chen, and L. Shi. 2017. “A new budget allocation framework for the expected opportunity
cost”. Operations Research 65(3):787–803.
Gao, S., J. Du, and C.-H. Chen. 2019. “Selecting the Optimal System Design under Covariates”. In
Proceedings of the 15th IEEE International Conference on Automation Science and Engineering,
547–552. IEEE Press.
Gao, S., H. Xiao, E. Zhou, and W. Chen. 2017. “Robust ranking and selection with optimal computing
budget allocation”. Automatica 81:30–36.
Gelman, A., J. B. Carlin, H. S. Stern, D. B. Dunson, A. Vehtari, and D. B. Rubin. 2014. Bayesian Data
Analysis. CRC Press.
Kim, E. S., R. S. Herbst, I. I. Wistuba, J. J. Lee, G. R. Blumenschein, A. Tsao, D. J. Stewart, M. E. Hicks,
J. Erasmus, S. Gupta et al. 2011. “The BATTLE trial: personalizing therapy for lung cancer”. Cancer
discovery 1(1):44–53.
Kim, S.-H., and B. L. Nelson. 2001. “A fully sequential procedure for indifference-zone selection insimulation”. ACM Transactions on Modeling and Computer Simulation 11(3):251–273.
Kim, S.-H., and B. L. Nelson. 2006. “Selecting the best system”. Handbooks in Operations Research andManagement Science 13:501–534.
Li, X., X. Zhang, and Z. Zheng. 2018. “Data-driven ranking and selection: high-dimensional covariates
and general dependence”. In Proceedings of the 2018 Winter Simulation Conference, 1933–1944. IEEE
Press.
Liu, A., Y. Zhang, and J. Li. 2009. “Personalized movie recommendation”. In Proceedings of the 17th
ACM international conference on Multimedia, 845–848. ACM.
Luo, J., L. J. Hong, B. L. Nelson, and Y. Wu. 2015. “Fully sequential procedures for large-scale rankingand-selection problems in parallel computing environments”. Operations Research 63(5):1177–1194.
Peng, Y., C.-H. Chen, M. C. Fu, and J.-Q. Hu. 2016. “Dynamic sampling allocation and design selection”.
INFORMS Journal on Computing 28(2):195–208.
Peng, Y., E. K. Chong, C.-H. Chen, and M. C. Fu. 2018. “Ranking and Selection as Stochastic Control”.
IEEE Transactions on Automatic Control 63(8):2359–2373.
Rinott, Y. 1978. “On two-stage selection procedures and related probability-inequalities”. Communicationsin Statistics - Theory and Methods 7(8):799–811.
Shen, H., L. J. Hong, and X. Zhang. 2017. “Ranking and selection with covariates for personalized decision
making”. arXiv preprint arXiv:1710.02642.
Zhang, S., Z. Jin, and J. Zhang. 2016. “The dynamical modeling and simulation analysis of the recommendation on the user–movie network”. Physica A: Statistical Mechanics and its Applications 463:310–319.
AUTHOR BIOGRAPHIESHAIDONG LI is a Ph.D. candidate in the Department of Industrial Engineering and Management, Peking
University, Beijing, China. He received his B.S. Degree from the Department of Engineering Mechanics
at Peking University. His research interests include simulation optimization and network analysis. He
is a visiting scholar under the supervision of Prof. Henry Lam during 2019-2020. His email address ishaidong.li@pku.edu.cn.
HENRY LAM is an Associate Professor in the Department of Industrial Engineering and Operations
Research at Columbia University. He received his Ph.D. degree in statistics from Harvard University in
2011, and was on the faculty of Boston University and the University of Michigan before joining Columbia
in 2017. His research focuses on Monte Carlo simulation, uncertainty quantification, risk analysis, andstochastic and robust optimization. His email address is henry.lam@columbia.edu.

Li, Lam, Liang, and PengZHE LIANG is a Professor in the Department of Management Science and Engineering, Tongji University, Shanghai, China. He received the B.E. degree in Computer Engineering from National University ofSingapore, Singapore, Singapore, in 2001, and the Ph.D. degree in Industrial and Systems Engineeringfrom Rutgers, The State University of New Jersey, New Brunswick, United States, in 2011, respectively.
His research interests include combinatorial optimization, mathematical programming, and scheduling andmanagement in aviation, logistics, ports, transportation, and telecommunications. His email address isliangzhe@tongji.edu.cn.
YIJIE PENG is an Assistant Professor in the Department of Management Science and Information Systems
in Guanghua School of Management at Peking University, Beijing, China. He received the B.S. degree inmathematics from Wuhan University, Wuhan, China, in 2007, and the Ph.D. degree in management sciencefrom Fudan University, Shanghai, China, in 2014, respectively. His research interests include stochastic
modeling and analysis, simulation optimization, machine learning, data analytics, and healthcare. His email
address is pengyijie@pku.edu.cn.