Recruitment Strategies That Take a Chance
Authors: Gregory Kehne
Ariel D. Procaccia
Jingyan Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: decision making under uncertainty
approximation algorithms
stochastic optimization

Abstract: In academic recruitment settings, including faculty hiring and PhD admissions, committees aim to maximize the overall quality of recruited candidates, but there is uncertainty about whether a candidate would accept an offer if given one. Previous work has considered algorithms that make offers sequentially and are subject to a hard budget constraint. We argue that these modeling choices may be inconsistent with the practice of academic recruitment. Instead, we restrict ourselves to a single batch of offers, and we treat the target number of positions as a soft constraint, so we risk overshooting or undershooting the target. Specifically, our objective is to select a subset of candidates that maximizes the overall expected value associated with candidates who accept, minus an expected penalty for deviating from the target. We first analyze the guarantees provided by natural greedy heuristics, showing their desirable properties despite the simplicity. Depending on the structure of the penalty function, we further develop algorithms that provide fully polynomial-time approximation schemes and constant-factor approximations to this objective. Empirical evaluation of our algorithms corroborates these theoretical results.

Recruitment Strategies That Take a Chance
Gregory Kehne
Ariel D. Procaccia Harvard University
Jingyan Wang
Harvard University
Georgia Institute of Technology
Abstract
In academic recruitment settings, including faculty hiring and PhD admissions, committees aim to maximize the overall quality of recruited candidates, but there is uncertainty about whether candidate would accept offer if given one. Previous work has considered algorithms that make offers sequentially and are subject to a hard budget constraint. We argue that these modeling choices may be inconsistent with the practice of academic recruitment. Instead, we restrict ourselves to a single batch of offers, and treat the target number of positions we risk overshooting or undershooting the target. Specifically, our objective is to select a subset of candidates that maximizes the overall expected value associated with candidates who accept, minus an expected penalty for deviating from the target. We first analyze the guarantees provided by natural greedy heuristics, showing their desirable properties despite the simplicity. Depending on the structure of the penalty function, we further develop algorithms that provide fully polynomial- time approximation schemes and constant-factie approximations to this objective. Empirical evaluation of our algorithms corroborates these theoretical results.
soft constraint,
Introduction
Anyone who has served faculty hiring committee or PhD admissions committee knows that successful outcome requires resolving the tension between two competing goals. On the one hand, some candidates are (perceived to be) better qualified than others, and the aim is to recruit the best candidates. On the other hand, there are a given number of positions to be filled, and while there is typically some flexibility, there is real cost to recruiting arises in part because the stronger candidate is, the more likely they are to receive multiple attractive offers and the less likely they are to accept any particular offer. In order manage uncertainty, good strategy may involve a mix of offers to stellar candidates and "safer" candidates.
many or too few people. The tension
To formalize this problem, assume that recruiting entity (academic otherwise) has access two numbers for each candidate i: their value H; and their probability p: of accepting an offer. We acknowledge that in current practice, these numbers committees typically rank or assign numerical scores to candidates based on their strength or fit, and savvy committees roughly estimate recruitment chances by classifying candidates as, say, "high yield," "low yield" or "extremely low yield", for example, by past experience or assistive computational tools [13, 1]. Therefore, we believe that the gap between current practice and explicit value and probability estimates is not large.
not always explicitly estimated. However,
Our approach builds the work of Purohit al. [12]. who cast hiring under uncertainty stochastic optimization problem. their basic model, there are n candidates (each associated with value and probability), k positions, and t time steps. In each time step, the algorithm (i.e., recruitment strategy) may make an offer to a single candidate and receive response; that is, at most t sequential offers can be made, and the budget of & cannot be exceeded. The goal is maximize the expected value of candidates who accept offers. Purohit et al. [12] also consider the setting where the algorithm may make parallel offers in each round. For both problems, they develop polynomial-time, constant-factor approximation algorithms (with approximations ratios and 8, respectively).
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

This problem formulation captures key aspects of recruitment, but, in our view, it does have two shortcomings. First, in sense it is overcomplicated, as computational challenges stem from the assumption that offers are made sequentially. But, in our experience of faculty hiring and PhD) admissions in several universities, offers are typically made in one batch. Indeed, delayed offers (in the case of faculty hiring) and waitlists (in the case of PhD admissions) are usually avoided as they negatively impact yield.
The second, and more crucial, shortcoming is that Purohit et al. [12] consider the constraint of hiring k candidates as firm. Again, this is inconsistent with our experience: offers are made so that the expected yield roughly matches a desired target, but some faculty hiring or PhD admission cycles are "too successful." in the sense that the number of candidates who accept their offers is much larger than expected. This has a real cost: in the case of PhD students, it creates difficulties in finding funding and advisors, and in the case of faculty hiring. it may precipitate a shortage of resources with long-term impacts on future hiring and even tenure. For example, in one of our institutions, a faculty hiring cycle with yield that was much higher than expected led to the cancellation of the subsequent year's search.
Let us, therefore, reformulate the problem of hiring under uncertainty in a way that avoids both issues. We assume that offers are made in a single batch, and the target number of positions k is a soft constraint. Specifically, a penalty is incurred for deviating from the target number of positions; we consider several different options for this penalty function. The optimization problem is this:
Select a subset S of candidates that maximizes overall expected reward Eies Pi -T minus expected penalty for deviating from the target number of positions.
Enumerating all possible subsets S may be practicable for small instances, for example in the case of faculty hiring in small departments. However, a brute-force approach will not work for this purpose in larger departments, or at the scale of PhD admissions even in smaller programs, which motivates our search for good algorithms.
Our results. We first consider a simplified case where the goal is to solely minimize the penalty term of our objective (irrespective of the rewards), and show that the greedy algorithm that selects candidates in decreasing order of their probabilities is optimal (Section 3.1).
The full objective is considerably more complex, and we analyze it under two natural penalty functions. When the penalty function is the squared error from the target, we show that the optimization problem is weakly NP-hard, and provide fully polynomial-time -ime approximation scheme (FPTAS). When the penalty is linear in the extent to which the target is exceeded (that is, a linear penalty is incurred by overshooting, but not by undershooting), we show that two greedy heuristics picking in the decreasing order of the value I; and the expected value p;7; provide approximations to the optimal solution that are polynomial in the minimum probability Pmin- We then present a constant-factor approximation algorithm that runs in polynomial time for fixed pmin and candidate value relative to overshooting penalty, thereby improving upon the greedy heuristics.
Finally, we carry out experiments on synthetically generated data (Section 4), focusing on the linear penalty incurred by overshooting. We observe that the two greedy heuristics perform reasonably well, especially if the values and probabilities are positively correlated. At the same time, compared to the greedy heuristics, our constant-factee approximation algorithm better adapts to specific instances, especially when this correlation is negative. These numerical experiments corroborate our theoretical results that the greedy heuristics provide reasonable guarantees, justifying their use in practice for both simplicity and good performance. However, in many regimes the additional flexibility of the constant-factor approximation algorithm is likely worth its complexity overhead.
Related work. In practice, the challenge of uncertainty in admissions is mitigated by practices such as admitting students in multiple rounds, using a waitlist, and using rolling process [9]. There are also assistive computational tools that predict student yield rate with machine learning [13, 1]. There are few theoretical formulations that model and address the uncertainty in such problems. As previously mentioned, Purohit et al. [12] consider an online setting where in each time step a candidate is given an offer then their decision is revealed immediately, and analyze the optimal ordering to give offers to the candidates subject to a hard constraint on the total number of acceptances. Ganguly et al. [7] consider a setting with multiple rounds where the yield rate in each round is either

---

7p or QH (with aL < 9H). To model the negative correlation between the candidate quality and the probability of acceptance, they assume that the probability of the yield rate being 9F is linear in the number of students given offers. They subsequently derive a decision tree that computes the number of offers to make in each round. For single batch selection, Zhang and Pippins [16] analyze the optimal number of applicants to admit using techniques from yield management, under the assumption that each applicant has identical value and probability. A distinct line work casts the admissions problem modeled by the students' stochastic preferences over multiple schools. An objective combining the utility and the accepted size is considered, but the penalty is in terms of the expected size, and does not consider variance. Another line of work analyzes various metrics for the secretary problem under uncertain offer acceptance [14, 15, 11], where candidates appear sequentially in a uniformly random ordering in a fleeting nature. Here the reward defined hased on the ranking of the candidate accepting the offer relative to other candidates in the sequence, and the goal is decide whether to make an offer to the candidate appearing in each time step.
decentralized matching market [4, 5J. where the uncertainty in acceptance
Our proposed formulation is closely related to the knapsack problem and its many variants. In the stochastic knapsack problem, each item has deterministic value and an independent stochastic size; the actual size of an item is revealed only after it is selected [8, 6, 2]. For the one-shot version of this problem, the aim is to choose a subset maximizing the expected value of the realized items, such that the probability that these realizations violate the knapsack constraint is below some threshold. In contrast, our "items" have equal size, but we pay a penalty which is some function of our realized distance from our knapsack target size, exchanging the constraint for mixed-sign objective. In this spirit, the one-sided loss functions we consider are similar to objectives which arise in the penalty method for solving constrained optimization problems [10], though setting is stochastic and do not introduce the penalty term in service of ultimately satisfying a hard constraint.
2 Problem Formulation
Taking knapsack perspective, consider some items (corresponding candidates) with associated values -1,-.., Iy E TR. If we select an item i E [n], there is a probability p: E [0, 1] that we receive this item (the candidate accepts the offer). We write these values and probabilities and p; E [0, 1] Let Zi E {0, 1) be the indicator variable that we receive item if it is selected, so that Z; Ber(p:). We assume the events that we receive each individual item are independent. Let Sz C S denote the random realization of chosen items; that is, Sz == (i E S Zi = 1}. Our goal is to select a subset S S [n]. First, we consider the reward for a subset S as the expected total value obtained:
vectors
IR"
R(S) = E Σ Z,
Σ P., iES
ES
At the same time, let M N denote a target size that want the realized set Sz achieve. We want to control the expected deviation of the realized size of Sz, which is |Sz] = Eies Zi. from the target size M. We consider this penalty
V(S) == E. ([Sz). M)].
where p: NxN, + Rgo is a loss function, to be specified later. Combining the two parts, we define the overall objective
U(S) R(S) - X- V(S),
where À E IR,, is a hyperparameter that governs the importance of the penalty relative to the reward. Our goal is to find the subset that maximizes the overall expected utility:
S E arg max US)..
SC(x]
We denote a problem instance by I := (x,, p, M, A), and the solution S* is thus a function of the problem instance and the loss function p. It is worth noting that since the overall utility U is mixed-sign objective, the optimal value of US*) may be negative depending on the instance and choice of p.
We consider a range of choices for the loss function p. Given a target size M, it is natural for p to be convex function minimized at M, which penalizes any deviation from M, or alternatively
3

---

monotone convex function that is nonzero above M, which can be seen as penalizing violation of budget constraint. We focus in particular on one- and two-sided linear and quadratic losses, which are formally introduced in Section 3.2 below.
3 Theoretical Results
To begin note that if only consider the reward term and set the penalty term to be V(S) == 0. then the solution is to trivially select all items. In what follows, we first discuss the other extremal case, taking R(S) == and considering the penalty V in isolation. These may be viewed as the extreme cases when = and -> oc. We will then turn to the general objective and consider both terms jointly.
3.1 Warm-Up: Penalty Only
To gain intuition for this problem, we start with the simplified case in which our goal is only to minimize the penalty term. Note that in this case our objective is strictly nonpositive.
Algorithm 1 PGREEDY
Require: p 1]
1: S - 0.
2: Sort (Paheix) in decreasing order and re-index the items such that p1 2 ... 2 Pn
3: for i 11=11,2 111,,2,..... do
4:
if US (i)) U(S) then
5:
SU
6: else
7: break
8:
return
We consider PGREEDY, the greedy algorithm with respect to p; (Algorithm 1). In words, PGREEDY selects items in their decreasing order of probabilities, with ties broken arbitrarily there are multiple items with the same probability.' ! The algorithm keeps selecting the next item defined by this order, and terminates when adding the next item would decrease the objective. This greedy algorithm is computationally efficient, since the stopping criterion in Line can be checked in polynomial time given access to (see Lemma 5 in Appendix B.8 for details). Surprisingly, PGREEDY is optimal for minimizing ((S) in isolation.
Proposition 1. Let M E N. be any target size. If the loss function MM) is convex, then PGRBEDY (Algorithu 1) yields optimal solution to minimizing the penalty mingg["] V(S).
The proof this proposition is provided in Appendix B.3. This result is not obvious, as one might expect that as the sum of probabilities of all selected items so far approaches the target size, it may be better to select an item with lower probability than an item with higher probability to "fill the gap." This is not true. Intuitively, it is because the realization of each acceptance Z, is binary, the outcome adding another item into the selection is either we add this item (with probability p.) or not (with probability 1 p:). If adding this item gives lower penalty, then we desire to add the item with the highest probability possible.
3.2 The General Objective
We now turn to the general objective. At the outset it bears noting that U(S) is submodular in S whenever p(-,M is convex (see Lemma in Appendix B. as is the case for the loss functions we consider. Unfortunately the existing body of work on mon-mmonttone) submodular maximization cannot be leveraged obtain a general-purpose approximation and may be negative even at optimality, and applying an affine transformation in order to engineer nonnegativity will generally destroy any approximation guarantees.
U(S). since U mixed-sign
We focus on a few natural choices of the loss function. First, we consider p given by linear and quadratic losses, which denote by L1 and L2 respectively. These yield penalty terms V(S) which
In practice it is natural to break ties in favor of items of higher value, though this does not affect results.

---

are equal to the mean average error (MAE) and mean squared error (MSE) for the realized size of the subset Sz. We also consider the corresponding one-sided losses, defined by
||21 - M 0
| M
I (152),
and (152) M) (1S2), M)².
otherwise,
All of these losses considered penalize the case where the realized size is greater than the target size M. In applications such as admissions and hiring, there is limited, pre-specified amount of resources allocated to the newly admitted or hired people. Hence, having more people than the target size is not desired. At the same time, the two-sided losses give explicit preference that the realized size should also not be smaller than the target size. This explicit penalty for undershooting could represent a hit to morale (an unsuccessful recruitment cycle really is demoralizing) or insufficient staffing for required tasks, such as teaching certain courses. The one-sided loss functions may still to some extent capture these considerations, as there is an implicit opportunity cost described by the reward term when fewer candidates accept.
3.2.1 An FPTAS for L2 Loss
Given that we understand our problem in both extremal cases (when only considering the reward term or the penalty term), one might hope that some interpolation between them could solve the general case. However, the general case is more complicated. Recall that for the penalty-only objective, PGREEDY in Algorithm attains the optimal selection, by adding items in decreasing order of pi and terminates once the next item strictly decreases the objective. But PGREEDY is clearly ill-suited to the general objective, since it does not take values into consideration. We now present two more natural greedy heuristics analogous to PGREEDY, and show that they are provably not optimal for the general objective. Specifically, we consider:
XGREEDY: adds items in decreasing order of their value n,, and terminates once the next item in this order strictly decreases the objective.
XPGREEDY: adds items in decreasing order of their expected reward x,p;, and terminates once the next item in this order strictly decreases the objective.
Despite these heuristics appearing intuitive, they perform a certain sense arbitrarily poorly even for the (squared) Le loss, as formalized by the following result.
Proposition 2. Consider the two-sided loss p L2 and any X O. Then for PGREEDY, XGREEDY and XPGREEDY, there exists an instance such that the algorithu selects S S [n]] for which U(S) S 0, while U(S*) > O.
The proof of this proposition is provided Appendix B.4, and we now provide an informal description of the instances constructed. Since PGREEDY does not take into account the item values x; at all, it is natural to expect that PGREEDY is not suitable for the general objective. Specifically, we consider two items where one item has probability 1 and value O, and the other item has "good" probability less than and a "good" value. Then PGREEDY selects the first item, whereas selecting the second item yields positive objective value. For XGREEDY and XPGREEDY, have almost the same expected reward. We let item has a probability of 1. We let item 2 have a slightly greater expected reward for tie-breaking, and let item 2 have a smaller probability. In this case, XGREEDY and XPGREEDY start by picking item 2, which introduces nontrivial variance. When becomes large, this variance drives the overall objective negative. On the other hand, picking item 1 yields a strictly positive objective.
consider two items that
Despite the failure of heuristic approaches, when the chosen loss function is p Tr our problem can in fact be approximated up to negligible additive error. For this loss, our full objective (1) may be written
U(S) Σ P.T. - A.E
Σ Zi M iES
(2)
EES
5

---

Letting by E {0, 1 be the binary decision variable for whether i E S, that is, 6, 11i E S}. the
optimization problem then becomes
2
arg max U(S) arg max SE(N]
b,p,2, - A-E E iE(M)
Σ b;Z; - M (E(m)
(3)
(E(0,1)"
Expanding (3) yields a collection of terms which are constant, linear, and quadratic in b, and so the objective can be reformulated an unconstrained binary quadratic program (UBQP). Although UBQP is strongly NP-hard, Çela et al. [3] present a pseudo-polynomin time algorithm for UBQP when the coefficient matrix for the quadratic form in the objective has constant rank. By proving that the objective (2) is sufficiently insensitive to small changes in our problem parameters, we leverage this secudo-polnnomia time algorithm to derive a FPTAS for approximating the optimal objective value for our problem. A standard search-to-decision reduction then yields the following result.
Theorem 1. For p L2, Algorithm 4 identifies some S S [n] satisfying U(S) N U(S*) in time poly(1/e, n, M, X).
Pseudocode describing Algorithm 4 and the proof of this theorem are provided in Appendix B.5. On the other hand, by a reduction from equipartition we have the following hardness:
Theorem 2. For = Lz2, optimizing (S) is weakly NP-hard.
The proof of this theorem is provided in Appendix B.6. The hardness landscape of our problem when p == L2 is therefore similar to that of the knapsack problem, which is heartening since the knapsack problem is relatively tractable in practice. However, unlike the knapsack problem, we should only hope for additive rather than multiplicative guarantees; this is because for = L2 our optimal value and may be strictly negative, even if all values z; are nonnegative.
not bounded away from
In contrast to 12, we find that the one-sided loss L. is not straightforward to analyze. In this case the objective does not admit a quadratic factorization in terms of decision variables, and although the objective is nonnegative, is difficult to analyze the performance of the greedy algorithm or contend with the nonlincarity of the loss function in principled way. We instead turn to L loss, where surprisingly these obstacles can be overcome.
3.2.2 Approximations for L Loss
The loss p L enables the possibility of multiplicative approximation because the optimum value U(S*) of the mixed-sign objective is nonnegative. More generally, any P with p(0, M) = - has nonnegative optimal value, since in this case (()) = O. For = I.f choosing any single item with positive x; and p; has strictly positive objective, since M 2 1 implies that I (1, .M = 0 and so U((i)) = per; More generally, for this loss it suffices consider only i for which æ;; >> since under this loss the marginal contribution of any i with x; < 0 is nonpositive.
The L loss may appear amenable to greedy algorithmic approach, since early items incur no penalty and the marginal penalty of adding a later item simply turns out to be proportional to the probability that the current solution exceeds the target M. However, as in the case for the Lg loss, these natural heuristics fail to consider the relation between items in the selection. While efficient, these greedy algorithms perform arbitrarily badly compared the optimal solution in the worst case. The failure of PGREEDY is again apparent as in the case for the L2 loss. We now provide some informal "bad" instances for XGREEDY and XPGREEDY for intuition.
Recall that XGREEDY chooses items decreasing order by value. Consider M 1, and consider two types of items with (1), (1, p) and (.2, p2) (0.5, 1). XGREEDY picks item 1 and yields an objective of (p)), whereas picking item 2 yields constant objective. The other greedy algorithm XPGREEDY chooses items i in decreasing order of their expected value x,p;. We consider the instance with M = 1, and two types of items (1).P1) = (1 + e, 1) and (Z2.P2) = (1/p, p) with some tiny >> 0 so that XPGREEDY chooses item from type and yields constant objective. On the other hand, choosing 1 copies of item yields an objective of (1pp))
For both XGREEDY and XPGREEDY, the constructed instances yield an upper bound on the approxi- mation ratio, scaling as the minimum probability Pmin min4g(n] Pi associated with the items. It suggests that Pmin is natural parameter for measuring the complexity of an instance with respect to the L loss. Another natural parameter is Tmaa max (n] 24, the maximum value among all items.

---

Surprisingly, the performance of these greedy algorithms can be bounded in terms of Pmin as well, under an additional assumption about the values of items relative to A.
Theorem 3. Consider the one-sided loss p = E.-- If there is some fixed constant e > O such that
Tmax (1 -c c) X, then
(a) There exist instances for which PGREEDY selects S with U(S) = O, while U(S*)= > O.
(b) The worst-cuse approximation ratio for XPGREEDY is (pmmin).
(c) The worst-case approximation ratio for XGREEDY is (pmin) and ((min))-
The proof of this theorem is provided in Appendix B.7. As consequence, the approximation ratios of these greedy algorithms can be arbitrarily small as Pmin + O. Also note that the upper bounds do not require the assumption that 2; S (1 c) in all three cases there exist bad instances irrespective of this assumption. The reason we can derive better lower bounds for XPGREEDY than XGREEDY is intuitive; this is because XPGREEDY measures the expected reward conferred by an item, and so can be more directly related to the optimal solution S*
This is in notable contrast to when = L2g, where we saw that multiplicative guarantees are inapt and all three greedy algorithms may incur arbitrarily large additive loss. Theorem also raises a natural question: is this Pmin upper bound tight, or is it possible to efficiently attain better approximations to U (S*) which do not depend on Pmin? We address this by introducing ONESIDEDL; presented in Algorithm 2, which attains constant-factor approximation to the optimal solution. In this pseudocode, for any vector E IR" and set S - [n], we use els to denote the |S] dimeennsioa vector obtained by restricting u to its coordinates indexed by S. Its runtime is parameterized by the minimum probability Pmin and the ratio of the maximum value Tmax to the penalty parameter X; in particular for fixed Pmnin and X/Tmmax it is polynomial in m.
?
At a high level, ONESIDEDL;" proceeds first by dividing the items into three groups according to their values 24, and considering each group in turn. Since U is submodular (see Lemma 1 in Appendix B.1), the optimal solution within at least one of these groups is also constant-competitive with (S*). We obtain a constant-factor approximation for each group in different way.
For the items with low values bounded away from X, Low VALUEL (Algorithm 5 in Appendix B.8) checks all small subsets, which succeeds if the optimal subset for this group small. It also computes rounded probabilities and values for each item in this group, and then efficiently computes the optimal solution according to this rounded instance. If the optimal subset is large, we then prove that this search over rounded solutions necessarily identifies a subset with objective value comparable to that of the optimal subset. This is the technical crux of proving that ONESIDEDL; is a constant-factor approximation.
For the items with values just below X, MEDIUM VALUEL (Algorithm in Appendix B.8) returns the optimal subset if the group is small. If the group is large, it tries to choose a subset such that the expected number of realizations is about M: if there are not enough items, it chooses a subset with approximately half the expected number of realizations of the group overall. Finally, for the group of items with values above X, is straightforward to see that choosing the entire group is optimal. The pseudocode and related proofs for these algorithms appear in Appendix B.8. The following result provides a theoretical guarantee for Algorithm 2.
Theorem (Constant-factor approximation for Lt). Algorithm 2 is a constant-factor approximation of max{ 1.log (inu ).og (1og((2))) )))
to U(*) which runs in time n
The proof of this theorem is provided in Appendix B.8. Intuitively, the reason Algorithm 2 divides the items into cases depending on their values is to handle the case when the reward portion of US**) is almost equal to the penalty portion. This presents impediment to the performance of solving a rounded version of the instance, since in this case the magnitude and even the sign of U (S*) is potentially quite sensitive to changes in and z;. By restricting attention to items with low values, we prove that the expected number of realized items in S* not much more than the target M. This then allows to argue that there exist good rounded solutions that can be efficiently identified.
We conclude our theoretical results with surprising equivalence between one- and two-sided linear losses I.T. and Ln. In what follows, we use Uz, and UZ to denote the objective with p = T, and
7

---

Algorithm 2 ONESIDEDLI
Require: Problem instance I (r, M.
Ensure: 5 _ [n]] for which (S) (S*) ) for universal constant
I: Nz - {i E [n]: 22
<
(1
Pum }}
2: Nas - {i E [n]
24 < <}
3: Nи + E [n]: 24
4: Sr. < LOWVALUELL N2 PlN X, M)
5: SM < MEDIUMVALUEL (Z) Pl X, M)
6: Sμ <NH
7: Compute USL)) (SM). and USH
8: return S (Sr,, Sw, Sn} maximizing U(S)
P I, respectively. We U(; I) denote the evaluation of ((S) specifically with respect the instance = (x., P, A, M).
Theorem 5 (Equivalence between L1 and L For any instance I' = (x', L,',,,,, M) given by x = X and X' = 2X. Then for all S S [m].
(2. M). construct
UL (S: I) U L (S; I') - M.
The proof of this theorem provided in Appendix B.9. In particular, since and M do not depend on S, this implies that S maximizes Ut, on instance I if and only if it maximizes U UL! on instance T'
Although Theorem establishes a correspondence between the solutions to our problem for p= L1 and p LT, it is not approximation preserving, it does not convert ONESIDEDL; into an approximation algorithm for the two-sided setting. Indeed, as in the p = L2 setting, the optimal value when p L1 can be strictly negative.
Numerical Experiments
Having established worst-case theoretical guarantees, we wish to test how well our algorithms perform empirically. We focus on L loss because our result for L2 is an FPTAS, so we know its performance can be made arbitrarily close optimal. Specifically, the experiments benchmark the subroutine Low VALUEL," (part of ONESIDEDL;") against XGREEDY and XPGREEDY for the regime where 2; I (1 - c) This the regime which VALUEL! was developed to handle for ONESIDEDL;"- and is the regime for which we prove performance guarantees for XPGREEDY and XGREEDY. In Appendix A, we also provide comparison to the optimal solution for smaller instances (Appendix A. I). and compare XGREEDY to XPGREEDY under other losses (Appendix A.2). All error bars shown in the plots represent standard error of the mean. The code to reproduce simulation results is available at https: htttt::://gthubb com/ wttgya////grruttttaaaaee tment http://eeeerettttttttmmmrraaaaa
4.1 Experimental Setting
In constructing instances follow the approach of Purohit et al. [12] in their of beta distributions to orchestrate different kinds of correlation between x; and pt. We therefore first draw o; ~ Unif 1), and then produce three types of correlation follows:
Negative correlation: Pt Punin + (1 Puni) Beta - -x,,, 10xz).
Positive correlation: ~ Pmin + (1 Pmin) Beta(10z; 10(1 - xz)).
No correlation: P4 Unif[ 1].
This construction differs from the sampling paradigm Purohit et al. [12] only that we re- normalize the probabilities {p:) so that they are bounded in [Pmin: 1]. We consider n 50 and Pimin = 0.01 throughout, and explore greedy heuristics XGREEDY and XPGREEDY as well as the constant-factor approximation algorithm ONESIDEDL," CaaaonLLAAAAggiiii (Algorithm 2), for range of M and X.

---

Low +
3.5
LowValeel++
LowValuel1+ xGreedy xpGrredd
xGreedy SpGrordy
xGreedy spCreody
Objective
Obgeetive
Objective
1.5
1.0
2
1.0
0.5
Torget
targes
target
Figure 1: Sampling from the beta distribution with positive, no, and negative correlation. Here 72 50 and X
lambda= 1.5
lamhda=5
lambda=30
2.0
LowVsluel.1+
Valne
LowValuel.1++
xCreedy
Xrready xpGreedy
xCreody
2.0
xpGreedy
1,5
15
xpGreedy
Objertie
Objjective
Obijorton
1.5
1.0
0.5
0.5
0.0
Figure 2: Performance for increasing penalty regularizer A. Here n 50 and sampling via the negatively correlating beta distribution.
This lower bound on Pmin ensures that the performance of the greedy heuristics and runtime of our algorithm are reasonable; a value of 0.01 (say) is realistic because in practice, if a candidate takes the time and effort apply, it reasonable assume that they least have some nontrivial probability to accept if they were given offer. We also focus on the regime where z; < A, which is assumed by Theorem and handled in Algorithm by the subroutine Low VALUEL We believe that this the main regime of practical interest: candidates with z; 2 À are beneficial regardless of how many candidates have already accepted offers, and one might suppose that such candidates are rare.
Note that our theoretical guarantees in Theorem 4 necessitate that all candidate solutions to size Ö(1 /r are checked by brute force. In this implementation of Low VALUEL we take 0 and isolate its search over rounded solutions. LOWVAL.ULL, from outperforming the greedy algorithms simply by virtue of having considered every relevant solution. This only hinders the performance of Low VALUEL,+ This implementation additionally only considers rounded solutions satisfying Eies J.P; S 2M, which improves its
we consider small target sizes M, this prevents
runtime and theoretically only hinders its performance relative to Low VALUEL... So long as z; < and small solutions checked, such stopping condition can be implemented without hindering the performance of Low VALUEL;;; ;t for details, see Appendix B.8 and Lemma 2 in Appendix B.I. We believe this provides favorable tradeoff between runtime and accuracy, and illustrates lower bound on the performance of VALUEL as written.
4.2 Experimental Results
The objective values that our algorithms of interest attain these distributions are shown Figure 1. Note that positive correlation leads XGREEDY and XPGREEDY pursue very similar (and optimal)

---

strategies, as expected. This is intuitively the easier setting, and here Low VALUELL performs on par with the greedy heuristics. In the no-correlation and negative-correlation settings, there are regimes where one of the two greedy heuristics performs better than the other one, whereas LOW appears to perform as well as the better of two depending on the regimes, showing its better adaptivity across these instances in practice as well as in theory.
Negative correlation between I; and p; is of particular interest to us, since it seems most relevant for the setting of faculty hiring and PhD admissions, and in fact hiring and recruitment more broadly. In Figure 1, we also observe that negative correlation the setting that displays the most heterogeneity in algorithm behavior. We therefore turn to this
setting and explore the effect of increasing the penalty regularizer in Figure 2. general, Low VOVVALL appears comparable to the better of the two greedy heuristics across the values of M and À that we examine, though there is a small gap between the objectives achieved by LOWVALUEL. and XGREEDY when À 1.5.
This is also good news for XGREEDY and XPGRERDY, because it suggests that the two of them together remain competitive across wide range of instances. To the extent that LOWVALUEL, falls short of the objective achieved, it is due to systematically rounding the probabilities by constant factor when computing the prospective utility of solutions. Because its rounding preserves the reward term, such a systematic overestimate in p; leads it to overestimate the penalty term of any set under consideration. The impact of rounding may be small on each individual item but collectively large on the objective, and therefore explains the extent to which Low VALUEL, lags behind XGREEDY in Figure 2: the latter chooses many such items while the former judges their influence on the penalty to be too large. However, this can be mitigated by choosing smaller multiplicative bucket sizes for Low VALUEL. in rounding, which is particularly effective in the case where the probabilities {pi) of an instance fall in a small number of clusters or exhibit other structure.
up
5 Discussion
One of the takeaways from our theoretical and empirical results is that, in addition to XPGREEDY, the greedy algorithm XGREEDY, which makes offers to subset of candidates with the highest values, is practicable for Li loss. This is intriguing because the algorithm is quite similar to how faculty hiring and admissions committees typically think: they want to make offers to the best candidates. The difference is that XGREEDY carefully selects the number of offers to be made, in a way that greedily maximizes the objective. Since XGREEDY amounts relatively small tweak to current practice, believe committees would find the algorithm to be especially palatable.
An issue our results do not address is which penalty function best matches the needs of a specific recruitment process. For example, is there a rigorous way to argue that a particular choice of penalty function is more broadly applicable than another? That said, the choice between one-sided and two- sided penalty is rather intuitive, depending on the application. And our results provide computational arguments in favor of L2 when two-sided penalty is desired, and L for one-sided penalty.
From an ethical viewpoint, a potential concern is that our proposal may ultimately have unintended negative consequences. For example, if many faculty hiring committees adopted our optimization- based approach, might candidates have fewer opportunities? We believe, however, that the opposite is true. Currently the academic job market is strikingly inefficient, as committees often converge on few candidates who are inundated with interviews and offers, while comparably strong candidates are left with nothing. If our approach is adopted widely, it is likely to widen the pool of candidates who receive appealing offers. Granted, a centralized matching market (in the style of the National Resident Matching Program) may be an even better solution, but creating such market requires huge- and often impractical- -eeiiical-deeeee degree of coordination; by contrast, our approach can be adopted independently by institutions and even by individual departments committees.
Acknowledgments. We thank Anupam Gupta and Alejandro Toriello for helpful discussions. GK and AP were partially supported by the National Science Foundation under grants IIS-2147187, CCF. 2007080, 202-2024 and 173-5633;56; and by the Office of Naval Research under grant N00014- 20-1-2488. JW was supported by the Ronald J. and Carol T. Beerman President's Postdoctoral Fellowship and the ARC (Algorithms & Randomness Center) Postdoctoral Fellowship at Georgia Tech.
10

---

References
[1] John Abelt, Daniel Browning, Celia Dyer, Michael Haines, Jalen Ross, Patrick Still, and Matthew Gerber. Predicting likelihood of enrollment among applicants to the UVa undergraduate program. In Systems and Information Engineering Design Symposium, pages 194-199, 2015.
[2] Anand Bhalgat, Ashish Goel, and Sanjeev Khanna. Improved approximation results for stochas- tic knapsack problems. In Proceedings of the Twenty-Second Annual ACM-SIAM Symposium on Discrete Algorithms, pages 1647-1665, 2011.
[3] Eranda Çela, Bettina Klinz, and Christophe Meyer. Polynomially solvable cases of the constant rank unconstrained quadratic 0-1 programming problem. J. Comb. Optim., 12(3):187-215, 2006.
[4] Yeon-Koo Che and Youngwoo Koh. Decentralized college admissions. Journal of Political Economy, 124(5): 1295 1338, 2016.
[5] Xiaowu Dai and Michael I. Jordan. Learning strategies in decentralized matching markets under uncertain preferences. Journal of Machine Learning Research, 22(260):1-50, 2021.
[6] B.C. Dean, M.X. Goemans, J. Vondrák. Approximating the stochastic knapsack problem: The benefit of adaptivity. In 45th Annual IEEE Symposium Foundations of Computer Science, pages 208-217, 2004.
[7] Subhamoy Ganguly, Ranojoy Basu, and Viswanathan Nagarajan. A binomial decision tree to manage vield-uncertainty in multi-round academic admissions processes. Naval Research Logistics (NRL), 69(2):303-33 (122)3333311,20000
[8] Jon Kleinberg, Yuval Rabani, and Éva Tardos. Allocating bandwidth for bursty connections. SIAM Journal on Computing, 30:191-217, January 2000.
[9] Yao Luo and Yu Wang. Dynamic decision making under rolling admissions: Evidence from U.S. law school applications. Technical Report tecipa-681, University of Toronto, Department of Economics, 2020.
[10] Anne L. Olsen. Penalty functions and the knapsack problem. In Proceedings of the First IEEE Conference on Evolutionary Computation, IEEE World Congress on Computational Intelligence, June 27-29, 1994, pages 554-558. IEEE, 1994.
[11] Sebastian Perez-Salazze Mohit Singh, and Alejandro Toriello. Robust online selection with uncertain offer acceptance. CoRR, abs/21 12.00044, 2021.
[12] Manish Purohit, Sreenivas Gollapudi, and Manish Raghavan. Hiring under uncertainty. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 5181-5189, 2019.
[13] Linda Siefert and Fred Galloway. A new look at solving the undergraduate yield problem: The importance of estimating individual price sensitivities. College and University, 81(3):11-17, 2006.
[14] M. H. Smith. A secretary problem with uncertain employment. Journal of Applied Probability, 12(3):620 624-244,. 1975.
[15] Mitsushi Tamaki. A secretary problem with uncertain employment and best choice of available candidates. Operations Research, 39(2):274 284, 1991.
[16] Li Zhang and John C. Pippins. Application of overbooking model college admission The Citadel, International Transactions in Operational Research, 28(5):2402-2413, 2021.

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're rou're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
12


Reviews:
[{"review_id": "v6LindN6rao", "reviewer": "Reviewer_VdAs", "summary": "The paper considers a recruitment problem in which a agent can make offers to a set of possible candidates. Each candidate is characterized by a value and a probability of accepting the offer. The agent must make a single batch of offers, choosing a subset of the possible candidates. The objective is to maximize the expected value associated to the candidates that accept, minus a penalty term the depends on the deviation from a target number of desired acceptances.\nThey consider different types of deviations: $L_1$ and $L_2$ losses, and $L_1^+$ and $L_2^+$ losses in which recruiting less candidates that the target number is not penalized.\nThe authors characterized the computational complexity of computing optimal strategies and provide approximation results when the problem is intractable.\nFinally, they perform an experimental analysis. ", "questions": "No questions.", "limitations": "Yes", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper introduces a new computational problem. A similar problem was studied with sequential interaction, while this paper considers a different interaction with a single batch of  offers. This model is more realistic in many settings.\n\nThe paper is well written, easy to follow, and provides a detailed analysis of previous works.\n\nThe technical results are technically involved and provide a complete picture of the problem.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "qtWb9XwPs3o", "reviewer": "Reviewer_6MKo", "summary": "The paper considers a problem of academic recruitment where the committee aims to select a set of candidates that maximizes the total expected reward minus the penalty caused by deviation from the target (i.e., over-hiring or under-hiring). It considers three simple greedy algorithms: (1) PGREEDY: selecting candidates in order based on their likelihood to accept the offer $p_i$; (2) XGREEDY: selecting candidates in order based on their value (qualification) $x_i$; (3) XPGREEDY: selecting candidates in order based on $x_i\\cdot p_i$. The authors conduct the accuracy analysis of these greedy algorithms under certain settings and show both positive and negative results. For different penalty functions (i.e., one-sided and two-sided linear/square loss), the paper also proposes algorithms that attain fully polynomial-time approximation and constant-factor approximation to the optimal objective. ", "questions": "\n1. In lines 215-218, the authors explained briefly why it’s difficult to analyze the algorithms under one-sided L2+ loss. Can you elaborate more? What does “quadratic factorization” mean and why it is an important property the objective needs to have? Why linearity is necessary for analyzing the one-sided loss?   \n\n2. In experiments, the authors construct synthetic data based on the approach of Purohit et al. Can you explain why this is good modeling? How will the algorithms perform on data with other distributions?\n\n3. The experiments show that when $x$ and $p$ are negatively correlated, the performance of all algorithms drops significantly compared to positive and no correlation (Figure 1). Does it suggest that algorithms should be developed by incorporating such a negative correlation?\n\n4. In practice, penalty functions may deviate from one-sided linear functions. While the performance of algorithms is theoretically guaranteed under a one-sided linear function, they can still be evaluated empirically on other types of loss functions. I suggest authors conduct experiments on settings with other loss functions to examine the robustness of the algorithms. \n\n5. Algorithm 2, missing “{“ in lines 1-3; what are the notations $x|N_L$, $p|N_L$?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n1. The paper studies an interesting problem. In practical academic recruitments, multiple offers are typically made in parallel and target number of positions is typically treated as a soft constraint. This is captured in the model considered in this work.\n2. The paper considers several simple greedy algorithms and conducts theoretical analysis. These results provide insights for better hiring practices. \n\n\nWeaknesses:\n\n1. My biggest concern is the restrictive conditions/assumptions of the results. The analytical results highly rely on the structure of penalty functions (i.e., induced cost as a function of the target and the actual number of hires). The paper primarily considers two types of penalty function (one-sided linear loss and two-sided square loss), and most results and algorithms are limited to these function types. In practice, we don’t know the exact penalty function, and we see from the results that different penalty functions may lead to completely different conclusions. As a result, the proposed method may not be easily deployed in practice. \n2. The experiments only validate the proposed algorithms in synthetic settings; there is no real-world data that can be used for validation. Moreover, there is no other baseline method to compare with. The experiments show that when the candidate’s value $x_i$ and probability $p_i$ are negatively correlated (a realistic setting), the accuracy seems to be very low (e.g., in Figure 1, accuracy drops by almost half compared to the positive correlation case). It’s not clear how serious such a decrease is. What is the best attainable objective value $U(S*)$ for each target? It is important to highlight $U(S*)$ in the figures.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "drxf5aJf8Sg", "reviewer": "Reviewer_xufP", "summary": "The paper investigates the problem of batch hiring in an academic recruitment setting. While previous works consider a hard budget constraint on the number of acceptance, the authors argue that this assumption is inconsistent in practice and carries computational hardness. Instead, they view the budget as a soft constraint with a penalty for overshooting the target. Under these assumptions, the paper shows that greedy heuristics demonstrate some desirable properties. Finally, the authors provide an algorithm that approximates the optimal solution within a constant factor in polynomial time.", "questions": "- While the greedy heuristic is a natural benchmark, is there any other algorithm/benchmark that can be used for comparison in this setting? \n\n- Although the result for L_2 loss is discussed, does the same result hold for the two-sided L_1 loss? ", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact in their paper. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper provides a novel algorithm for the case of one-sided L_1 loss between the realized outcome and the target number of hires.\n\n- The paper is well-written and well-organized. \n\n- The related work section is sufficient. \n\n- The discussion of greedy heuristics performances is sufficient and provides good baseline intuition for the algorithm.\n\n- The theoretical guarantee of algorithm ONESIDEDL1+ is significant and improves upon the greedy heuristics in experiments. \n\n- The theoretical claims are supported by discussions that provide intuition to the proof.\n\n- The numerical experiments section contains a sufficient discussion of the setting and explanations for choices of hyperparameters. The results of these experiments support the theoretical claims made. \n\nWeaknesses:\n\n- There are a few inconsistencies in the numerical experiments section. In the description of Figure 1, the paper stated \"n=50\". However, in line 304, at the end of the line, the paper stated \"n=40\". In line 337, \"\\lambda = 1\", there is no plot for such value. Instead, in Figure 2, there are plots for $\\lambda$ with values 1.5, 5, and 30.\n  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ENMtGjIo-BH", "reviewer": "Reviewer_vVfn", "summary": "This paper proposes strategies to find the optimal batch of offers to make in a recruitment scenario, where the goal is to maximize the quality of the recruited candidates but there is a soft constraint on the maximum number of candidates that can be recruited (and it is uncertain whether candidates will accept an offer or not). When the penalty function that characterizes the soft constraint is based on the mean squared error, the authors provide a polynomial-time strategy that can be made arbitrarily close to being optimal; when the penalty function is one-sided and linear, they provide a polynomial-time strategy that has a constant approximation factor.", "questions": "In all experiments presented, xGreedy seems to be equivalent to, if not better than, LowValueL1+. What is the reason behind this result? Is it because LowValueL1+ was run with $\\tau=0$? If so, how sensitive is LowValueL1+ to the choice of $\\tau$ and how would one tune $\\tau$ for their specific problem instance? Is it because $p_{\\text{min}}$ is not low enough to see the advantage of LowValueL1+ being a constant-factor approximation (as opposed to xGreedy whose performance depends on $p_{\\text{min}}$)? If so, how does the performance difference between xGreedy and LowValueL1+ varies with respect to $p_{\\text{min}}$? Since the performance xGreedy is lower bounded in terms of $p_{\\text{min}}$, I assume it should be possible to make LowValueL1+ arbitrarily better than xGreedy. Is it possible to show this empirically? I believe more experiments are needed to answer these questions.\n\nAlso, I think evaluating the performance of the strategy proposed for $L_2$ with respect to the greedy strategies would have still been valuable. I understand that $L_2$ can be made arbitrarily close to being optimal but how would that compare to the greedy strategies? LowValueL1+ had different theoretical guarantees that made it more preferable over the greedy strategies, but the experiments revealed that xGreedy could perform better than a practical implementation of LowValueL1+ for most problem instances considered.", "limitations": "One limitation is that the strategies proposed in the paper are specifically designed for $L_2$ or $L_1^{+}$ penalty functions whereas a user might want to design their own penalty function according to the characteristics of their own problem instance based on historical data.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The problem seems to be well motivated. I especially like the explanation of why the sequential and hard-constrained setting studied previously in the literature fails to capture the more practical complexities of recruitment (as opposed to single-batch and soft-constrained setting considered in this paper). However, most of the evidence for this seems to be anecdotal, maybe it could be supported with more references.\n\nThe problem is explored fully. In particular, a practical strategy is proposed for both a two-sided penalty function (where recruiting fewer candidates than targeted is penalized explicitly) as well as a one-sided penalty function (where recruiting fewer candidates is not penalized explicitly, but it would implicitly lower the total quality of the recruited candidates). However, it is not discussed when would one strategy/penalty function would be preferred to the other. While a one-sided penalty function seems natural to me, it would have been nice to have examples where under-recruiting would require explicit penalty (on top of the opportunity cost of not recruiting as many candidates as possible).\n\nThe experiments seem to be the weakest part of the paper; I have put my questions regarding the experiments below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept