SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Implementation	Gurobi	Flow (w = 10)
Algorithm	PLRA	PM-Q	PM-E	PM-Q	PM-E
Wall Clock Time	1.35s	4.06s	17.64s	20.03s	20.63s
Total CPU Time	9.91s	31.13s	79.20s	20.03s	20.61s
User CPU Time	4.94s	23.46s	66.44s	20.03s	20.61s
System CPU Time	4.97s	7.67s	12.76s	0.00s	0.00s
Quality	95%	95%	95%	95%	95%
PQuality	N/A	1232.91	312.78	1230.75	312.22
Maxprob (↓)	0.80	0.80	0.80	0.80	0.80
AvgMaxp (↓)	0.80	0.74	0.74	0.74	0.74
Support (↑)	2501	28108	28099	5849	5853
Entropy (↑)	531.40	1953.55	1953.20	1411.82	1411.12
L2Norm (↓)	37.33	32.33	32.34	32.66	32.68
Table 2: Comparison of different implementations on AAMAS 2015. Downarrows  $(\downarrow)$  mean the lower the better and uparrows  $(\uparrow)$  mean the higher the better. Due to parallelization, Gurobi takes less wall clock time. However, the network-flow-based approximation uses fewer computation resources (total CPU time). The approximation has very accurate Quality and PQuality. Approximated Support and Entropy are significantly worse than Gurobi's, but still significantly better than PLRA's.
Implementation	Gurobi	Flow (w = 10)
Algorithm	PLRA	PM-Q	PM-E	PM-Q	PM-E
Wall Clock Time	21.50s	86.85s	368.13s	563.24s	554.07s
Total CPU Time	78.41s	988.11s	3212.18s	562.77s	553.59s
User CPU Time	67.66s	906.72s	2957.54s	562.68s	553.42s
System CPU Time	10.75s	81.39s	254.64s	0.09s	0.13s
Quality	95%	95%	95%	94%	94%
PQuality	N/A	606.55	305.41	603.34	303.68
Maxprob (↓)	0.90	0.92	0.92	0.90	0.90
AvgMaxp (↓)	0.90	0.87	0.87	0.86	0.86
Support (↑)	3648	7122	7480	6250	6432
Entropy (↑)	595.25	1589.95	1630.80	1575.86	1607.20
L2Norm (↓)	47.83	42.49	42.45	42.21	42.18
Table 3: Comparison of different implementations on ICLR 2018. Downarrows  $(\downarrow)$  mean the lower the better and uparrows  $(\uparrow)$  mean the higher the better. Due to parallelization, Gurobi takes less wall clock time. However, the network-flow-based approximation uses fewer computation resources (total CPU time). The approximation has relatively accurate Quality, PQuality and randomness metrics.
Comparing the running time of Gurobi and Algorithm 1. As shown in Tables 2 and 3, Gurobi generally runs faster than Algorithm 1 in wall clock time. This is because it is a well-written commercial software that is able to parallelize over multiple cores on our server. However, the sum of user and system CPU time is smaller for Algorithm 1, which shows that Algorithm 1 uses fewer computation resources than Gurobi. Moreover, note that Gurobi takes significantly longer to solve PM-E than PM-Q, while Algorithm 1 is not affected. The reason for this is Gurobi only supports quadratic objective functions primitively. Therefore, to solve exponential objective functions, iterative approximation methods have to be used. In contrast, Algorithm 1 can support an arbitrary perturbation function without sacrificing running time.
Comparing the solution qualities of Gurobi and Algorithm 1. In the rows about Quality and PQuality in Tables 2 and 3, we can see that Algorithm 1 has almost the same Quality as Gurobi. The PQuality of Algorithm 1 is also close to Gurobi's. This shows that with  $w = 10$ , Algorithm 1 approximates PM in solution quality well, validating Theorem 1.
Comparing the randomness metrics of Gurobi and Algorithm 1. As shown in Table 2, for randomness metrics Maxprob, AvgMaxp, and L2Norm, the approximated solution by Algorithm 1 has a similar or identical performance to Gurobi on AAMAS 2015. For Support and Entropy, the approximated solution is significantly worse than Gurobi's, but it is still significantly better than PLRA's solution. As presented in Table 3, on ICLR 2018, the performances of Algorithm 1 and Gurobi on all randomness metrics are relatively close. This shows that Algorithm 1 mostly preserves PM's performance on randomness metrics. Although on discrete-valued datasets like AAMAS 2015, Support and Entropy can be affected, they are still better than PLRA's solution.
A.2 Hyperparameter Tuning
In this section, we will introduce the way in which we do hyperparameter tuning for PM. We start with a lemma that shows the monotonicity of the solution quality of PM-Q with respect to  $\beta$ .
Lemma A.1. For an input instance  $(n_p, n_r, \ell_p, \ell_r, \mathbf{S})$ , let the solution of  $PM$  with  $f_1(x) = x - \beta_1x^2$  be  $\mathbf{x}_1$  and the solution of  $PM$  with  $f_2(x) = x - \beta_2x^2$  be  $\mathbf{x}_2$  where  $\beta_1, \beta_2 \in [0,1]$ . Then
$$
\beta_ {1} <   \beta_ {2} \Rightarrow \operatorname {Q u a l i t y} (\mathbf {x} _ {1}) \geq \operatorname {Q u a l i t y} (\mathbf {x} _ {2}).
$$
Proof of Lemma A.1: Let  $\mathrm{SQuality}(\mathbf{x}) = \sum_{p}\sum_{r}x_{p,r}^{2}S_{p,r}$ . Then
$$
\mathrm {P Q u a l i t y} _ {f _ {1}} (\mathbf {x}) = \mathrm {Q u a l i t y} (\mathbf {x}) - \beta_ {1} \mathrm {S Q u a l i t y} (\mathbf {x}),
$$
$$
\mathrm {P Q u a l i t y} _ {f _ {2}} (\mathbf {x}) = \operatorname {Q u a l i t y} (\mathbf {x}) - \beta_ {2} \mathrm {S Q u a l i t y} (\mathbf {x}).
$$
As  $\mathbf{x}_1$  maximizes  $\mathrm{PQuality}_{f_1}(\mathbf{x})$  and  $\mathbf{x}_2$  maximizes  $\mathrm{PQuality}_{f_2}(\mathbf{x})$ , we have
$$
\operatorname {Q u a l i t y} \left(\mathbf {x} _ {1}\right) - \beta_ {1} \operatorname {S Q u a l i t y} \left(\mathbf {x} _ {1}\right) \geq \operatorname {Q u a l i t y} \left(\mathbf {x} _ {2}\right) - \beta_ {1} \operatorname {S Q u a l i t y} \left(\mathbf {x} _ {2}\right), \tag {1}
$$
$$
\operatorname {Q u a l i t y} \left(\mathbf {x} _ {1}\right) - \beta_ {2} \operatorname {S Q u a l i t y} \left(\mathbf {x} _ {1}\right) \leq \operatorname {Q u a l i t y} \left(\mathbf {x} _ {2}\right) - \beta_ {2} \operatorname {S Q u a l i t y} \left(\mathbf {x} _ {2}\right). \tag {2}
$$
Subtracting (2) from (1), and using  $\beta_{2} > \beta_{1}$ , we have
$$
\left(\beta_ {2} - \beta_ {1}\right) \operatorname {S Q u a l i t y} \left(\mathbf {x} _ {1}\right) \geq \left(\beta_ {2} - \beta_ {1}\right) \operatorname {S Q u a l i t y} \left(\mathbf {x} _ {2}\right) \Rightarrow \operatorname {S Q u a l i t y} \left(\mathbf {x} _ {1}\right) \geq \operatorname {S Q u a l i t y} \left(\mathbf {x} _ {2}\right).
$$
Then, from (1), we know that
$$
\operatorname {Q u a l i t y} \left(\mathbf {x} _ {1}\right) - \operatorname {Q u a l i t y} \left(\mathbf {x} _ {2}\right) \geq \beta_ {1} (\operatorname {S Q u a l i t y} \left(\mathbf {x} _ {1}\right) - \operatorname {S Q u a l i t y} \left(\mathbf {x} _ {2}\right)) \geq 0.
$$
Therefore,  $\mathrm{Quality}(\mathbf{x}_1) \geq \mathrm{Quality}(\mathbf{x}_2)$ .
With Lemma A.1, for a minimum requirement of quality  $\mathrm{Quality}_M$ , we can use binary search to find the largest  $\beta$  for PM-Q such that PM-Q with  $f(x) = x - \beta x^2$  gives an assignment with quality  $\geq \mathrm{Quality}_M$ . Formally, consider the following Algorithm 2.
Algorithm 2: Hyperparameter Tuning for PM-Q
For a given minimum quality requirement  $\mathrm{Quality}_M$  and a slackness  $\delta$ :
(1) Use binary search to find  $Q_{\mathrm{PLRA}}$ , the smallest  $Q$  with which PLRA gives a solution with quality  $\geq \mathrm{Quality}_M$ .
(2) Use binary search to find  $\beta_{\mathrm{max}}$ , the largest  $\beta$  such that PM-Q with  $Q = Q_{\mathrm{PLRA}} + \delta$  and  $f(x) = x - \beta^2$  gives a solution with quality  $\geq \mathrm{Quality}_M$ .
(3) Finalize the hyperparameters as  $(Q,\beta) = (\min \{Q_{\mathrm{PLRA}} + \delta ,1\} ,\beta_{\max})$
Algorithm 2 maximizes the parameter  $\beta$  while ensuring the produced assignment with the set of hyperparameters  $(Q,\beta)$  has quality at least  $\mathrm{Quality}_M$  and Maxprob at most  $Q_{\mathrm{PLRA}} + \delta$ . Note that any solution with quality  $\geq \mathrm{Quality}_M$  has a Maxprob  $\geq Q_{\mathrm{PLRA}}$ . Intuitively, Algorithm 2 is
trying to maximize the randomness while obtaining the minimum required quality and near-optimal Maxprob given the quality constraint.
Analogously, we use a similar method for tuning the hyperparameters in PM-E.
Algorithm 3: Hyperparameter Tuning for PM-E
(1) Use binary search to find  $Q_{\mathrm{PLRA}}$ , the smallest  $Q$  with which PLRA gives a solution with quality  $\geq \mathrm{Quality}_M$ .
(2) Use binary search to find  $\alpha_{\mathrm{max}}$ , the largest  $\alpha$  such that PM-E with  $Q = Q_{\mathrm{PLRA}} + \delta$  and  $f(x) = 1 - e^{-\alpha x}$  gives a solution with quality  $\geq \mathrm{Quality}_M$ .
(3) Finalize the hyperparameters as  $(Q,\alpha) = (\min \{Q_{\mathrm{PLRA}} + \delta ,1\} ,\alpha_{\max})$
Although we similarly apply binary search to PM-E as we have done to PM-Q, the monotonicity condition of Lemma A.1 does not hold in general for PM-E. For instance, consider the following Example 1. If we run PM-E with different  $\alpha$  and  $Q = 1$ , the solution quality is not monotonic with respect to  $\alpha$ . The results are shown in Fig. 5.
Example 1. In this example,  $n_p = n_r = 3$ ,  $\ell_p = \ell_r = 1$  and
$$
\mathbf {S} = \left( \begin{array}{c c c} 0. 4 & 0. 0 & 0. 6 \\ 0. 8 & 0. 6 & 0. 0 \\ 0. 8 & 0. 6 & 1. 0 \end{array} \right)
$$
Figure 5: The quality of PM-E on Example 1 with  $Q = 1$  and different  $\alpha$ . The figure shows that the solution quality of PM-E is not necessarily monotonic with respect to  $\alpha$ .
Nevertheless, such examples are rare in practice. We have examined thousands of examples to find Example 1. In fact, the monotonicity of solution quality with respect to  $\alpha$  can still be observed empirically in the datasets we used. Therefore, for the purpose of our experiments in Section 6, we will still use Algorithm 3 to tune the hyperparameters of PM-E.
In Algorithms 2 and 3, there are still two parameters to be fixed, namely  $\delta$  and  $\mathrm{Quality}_M$ . In Section 6,  $\mathrm{Quality}_M$  was already specified as  $\{80\%, 85\%, 90\%, 95\%, 98\%, 99\%, 99.5\%, 100\}$  of the maximum possible quality. It remains to specify  $\delta$ . For experiments in Section 6, we used  $\delta = 0$  for AAMAS 2015 and  $\delta = 0.02$  for ICLR 2018. In the rest of the section, we will show experiments about the performances of PM-Q and PM-E with different  $\delta$  values to justify our choice. In practice, conference program chairs can simply choose a small value for the slackness, such as  $\delta = 0.02$ .
(a) AAMAS 2015  $\delta$  vs AvgMaxp (↓)
(c) AAMAS 2015  $\delta$  vs Support (↑)
(b) AAMAS 2015  $\delta$  vs L2Norm  $(\downarrow)$
(d) AAMAS 2015  $\delta$  vs Entropy  $(\uparrow)$
Figure 6: The performances of PM-Q and PM-E on AAMAS 2015 with different  $\delta$ .  $\mathrm{Quality}_M$  is set to  $95\%$  of the maximum possible quality. Downarrows  $(\downarrow)$  mean the lower the better and uparrows  $(\uparrow)$  mean the higher the better. The figure shows that the elbow points of both algorithms in all four metrics are at  $\delta = 0$ . Therefore, we should set  $\delta = 0$  for AAMAS 2015.
(a) ICLR 2018  $\delta$  vs AvgMaxp (↓)
(c) ICLR 2018  $\delta$  vs Support (↑)
Figure 7: The performances of PM-Q and PM-E on ICLR 2018 with different  $\delta$ . Quality  $M$  is set to  $95\%$  of the maximum possible quality. Downarrows  $(\downarrow)$  mean the lower the better and uparrows  $(\uparrow)$  mean the higher the better. The figure shows that the elbow points of both algorithms lie in the range  $\delta \in [0.02, 0.04]$ . We have chosen  $\delta = 0.02$  in this range.
(b) ICLR 2018  $\delta$  vs L2Norm  $(\downarrow)$
(d) ICLR 2018  $\delta$  vs Entropy (↑)
As we increase  $\delta$ , we allow PM-Q and PM-E to find more random solutions (i.e., solutions that perform better on our randomness metrics) at a cost of larger Maxprob. To balance the gain and loss, we choose  $\delta$  that maximizes a linear combination of them. We call these  $\delta$  elbow points. Below, we show the choices of  $\delta$  graphically in Figs. 6 and 7.
As shown in Figs. 6 and 7, on AAMAS 2015, the elbow points of both algorithms in all four metrics are at  $\delta = 0$ . Therefore, we should set  $\delta = 0$  for it. On ICLR 2018, the elbow points lie in the range  $\delta \in [0.02, 0.04]$ . We have chosen  $\delta = 0.02$  in this range for it.
A.3 Experiments on More Datasets
In this subsection, we include additional experiment results on more 4 datasets. In particular, we include Preflib1 (00039-1 from [39],  $n_p = 54$ ,  $n_r = 31$ ), Preflib2 (00039-2 from [39],  $n_p = 52$ ,  $n_r = 24$ ), Preflib3 (00039-3 from [39],  $n_p = 176$ ,  $n_r = 146$ ) and AAMAS 2016 (00037-2 from [39],  $n_p = 442$ ,  $n_r = 161$ ). All of the four datasets are bidding data consisting of 4 discrete levels: "yes", "maybe", "no" and "conflict". We transform these 4 levels to similarities  $1, \frac{1}{2}, \frac{1}{4}$  and 0 as done in Section 6. The constraints are set as  $\ell_p = 3$ ,  $\ell_r = 6$  for Preflib1 and Preflib3. For Preflib2 and AAMAS 2016,  $(\ell_p, \ell_r)$  are set as (3, 7) and (3, 12) respectively for feasibility.
We use the same experimental setup and hyperparameter setting as in Section 6. The results are shown in Fig. 8. The results of these additional datasets are similar to those on AAMAS 2015 shown in Figs. 2a and 3. In particular, both versions of PM achieve exactly the same performance with PLRA on Maxprob while improving significantly on the other randomness metrics. This shows that the empirical observations in Section 6 are not dataset-specific and generalize to other datasets.
(a) Preflib1 Quality vs Maxprob  $(\downarrow)$
(b) Preflib2 Quality vs Maxprob  $(\downarrow)$
(c) Preflib3 Quality vs Maxprob  $(\downarrow)$
(d) AAMAS2016 Quality vs Maxprob  $(\downarrow)$
(e) Preflib1 Quality vs AvgMaxp  $(\downarrow)$
(f) Preflib2 Quality vs AvgMaxp  $(\downarrow)$
(g) Preflib3 Quality vs AvgMaxp  $(\downarrow)$
(h) AAMAS2016 Quality vs AvgMaxp  $(\downarrow)$
(i) Preflib1 Quality vs L2Norm  $(\downarrow)$
(j) Preflib2 Quality vs L2Norm  $(\downarrow)$
(k) Preflib3 Quality vs L2Norm  $(\downarrow)$
(1) AAMAS2016 Quality vs L2Norm  $(\downarrow)$
(m) Preflib1 Quality vs Support  $(\uparrow)$
(n) Preflib2 Quality vs Support  $(\uparrow)$
(o) Preflib3 Quality vs Support  $(\uparrow)$
(p) AAMAS2016 Quality vs Support  $(\uparrow)$
(q) Preflib1 Quality vs Entropy  $(\uparrow)$
(r) Preflib2 Quality vs Entropy  $(\uparrow)$
(s) Preflib3 Quality vs Entropy  $(\uparrow)$
Figure 8: The trade-offs between quality and five randomness metrics on Preflib1, Preflib2, Preflib3 and AAMAS 2016. Downarrows  $(\downarrow)$  indicate that lower is better and uparrows  $(\uparrow)$  indicate that higher is better. The figure shows that PM incurs no increase in Maxprob while significantly improves over PLRA on all other four metrics. The results are similar to those on AAMAS 2015 shown in Section 6.
(t) AAMAS2016 Quality vs Entropy  $(\uparrow)$
B Missing Proofs in Sections 4 and 5
(a) Algorithm 1 runs in  $O\left( {w \cdot  {\ell }_{p} \cdot  {n}_{p}^{2} \cdot  {n}_{r}}\right)$  time.
(b) Algorithm 1 returns an assignment with perturbed quality  $\mathrm{ALG} \geq \mathrm{OPT} - f\left(\frac{1}{w}\right)\sum_{p,r}S_{p,r}$ .
Proof of Theorem 1: For (a), consider implementing the maximum cost flow using Edmonds & Karp's Algorithm B in [37]. In Edmonds & Karp's algorithm, we will compute the shortest path on the residual graph each time we find an augmenting path. The number of times we augment is bounded by  $O(w \cdot \ell_p \cdot n_p)$ . For the shortest path part, Edmonds & Karp's algorithm applies a node potential trick that ensures the edge weights are non-negative throughout the execution, so that we can use Dijkstra's algorithm for shortest path. Although for each paper  $p$  and reviewer  $r$ ,  $v_p$  and  $v_r$  are connected by  $O(w)$  edges, only 2 of them need to be considered: the ones with the largest cost in both directions  $v_p \rightarrow v_r$  and  $v_r \rightarrow v_p$ . Therefore, the shortest path can be computed in  $O(n_p \cdot n_r)$  time. Step (6) of Algorithm 1 can be done in  $O(n_p \cdot n_r)$  time by maintaining running totals for each reviewer and paper load. And thus Algorithm 1 runs in  $O(w \cdot \ell_p \cdot n_p^2 \cdot n_r)$  time.
For (b), let  $\mathbf{x}^*$  be the optimal assignment such that  $\mathrm{PQuality}_f(\mathbf{x}^*) = \mathrm{OPT}$ . Consider another assignment  $\mathbf{x}'$ , where  $x_{p,r}' = \frac{1}{w} \lfloor w \cdot x_{p,r}^* \rfloor, \forall p \in \mathcal{P}, r \in \mathcal{R}$ . Note that  $\mathbf{x}'$  may not be a feasible assignment if  $\sum_r x_{p,r}' \leq \ell_p$  for some paper  $p$ , but any feasible assignment  $\mathbf{x}''$  where  $x_{p,r}'' \geq x_{p,r}'$  for all  $(p,r)$  will have perturbed quality at least  $\mathrm{PQuality}_f(\mathbf{x}')$ . Then  $x_{p,r}^* - x_{p,r}' \in [0, \frac{1}{w}]$ , and
$$
\mathrm {P Q u a l i t y} _ {f} \left(\mathbf {x} ^ {*}\right) - \mathrm {P Q u a l i t y} _ {f} \left(\mathbf {x} ^ {\prime}\right) = \sum_ {p, r} S _ {p, r} \cdot \left[ f \left(x _ {p, r} ^ {*}\right) - f \left(x _ {p, r} ^ {\prime}\right) \right] \leq f \left(\frac {1}{w}\right) \sum_ {p, r} S _ {p, r}. \tag {3}
$$
On the other hand,  $\mathbf{x}^{\prime}$  corresponds to a feasible flow for the maximum-cost flow problem in Step (5) of Algorithm 1. Therefore, ALG  $\geq$  PQuality  $f(\mathbf{x}^{\prime})$ . Together with (3), we have ALG  $\geq$  OPT -  $f(\frac{1}{w})\sum_{p,r}S_{p,r}$ .
B.2 Proof of Theorem 2
Proof of Theorem 2: Label the first  $p_1$  papers and  $r_1$  reviewers as group 1, the next  $p_2$  papers and  $r_2$  reviewers as group 2, the next  $p_3$  papers and  $r_3$  reviewers as group 3 and so on.
For convenience of language, for an assignment  $\mathbf{x}$ , define the total Quality of paper  $p$  to be  $\sum_{r} x_{p,r} S_{p,r}$  and the total PQuality $_f$  of paper  $p$  to be  $\sum_{r} f(x_{p,r}) S_{p,r}$ .
Define  $\mathbf{x}^*$  to be the assignment such that for each paper  $p$  and reviewer  $r$ ,  $x_{p,r}^{*} = \ell_{p} / r_{i}$  if they are in the same group  $i$  and  $x_{p,r}^{*} = 0$  otherwise. Note that  $x^{*}$  is feasible, because  $\ell_p / r_i\leq Q$  according to assumption (iii), each row of  $\mathbf{x}^*$  sums to  $\ell_p$  and each column of  $x^{*}$  sums  $\leq \ell_r$  according to assumption (ii). Intuitively,  $x^{*}$  is a generalized version of Fig. 1a on blockwise dominant matrices.
Claim B.2.1. For feasible assignment  $\mathbf{x}$ ,  $\mathrm{Quality}(\mathbf{x}^{*}) \geq \mathrm{Quality}(\mathbf{x})$ . The equality holds if and only if  $\mathbf{x}$  assigns paper  $p$  to reviewer  $r$  with positive probability only when they are in the same group.
Proof of Claim B.2.1: First,  $\mathrm{Quality}(\mathbf{x}^{*}) = \sum_{i=1}^{k} p_{i} r_{i} \cdot \ell_{p} / r_{i} \cdot A_{i,i} = \sum_{i=1}^{k} p_{i} \cdot \ell_{p} \cdot A_{i,i}$ .
On the other hand, recall that  $A_{i,i} > A_{i,j}, \forall j \neq i$ . For a paper  $p$  in group  $i$ ,  $p$  can only be assigned to  $\ell_p$  reviewers, so the total similarity is at most  $\ell_p \cdot A_{i,i}$ . Therefore,  $\mathrm{Quality}(\mathbf{x}) \leq \sum_{i=1}^{k} p_i \cdot \ell_p \cdot A_{i,i} = \mathrm{Quality}(\mathbf{x}^*)$ . The equality holds if and only if each paper in group  $i$  is assigned to  $\ell_p$  reviewers with  $A_{i,i}$  similarity, i.e.,  $\ell_p$  reviewers in the same group.
As PLRA is maximizing Quality and  $x^{*}$  is a feasible solution, we have the following corollary.
Corollary B.2.1.  $\forall \mathbf{x} \in \mathrm{PLRA}(Q)$ ,  $\mathbf{x}$  assigns paper  $p$  to reviewer  $r$  with positive probability if and only if they are in the same group.
Next, we consider the performance of PM.
Claim B.2.2.  $\mathrm{PM}(Q,f) = \{\mathbf{x}^*\}$ .
Proof of Claim B.2.2: First,  $A_{i,i} \geq \operatorname{Dom}(\mathbf{A}) A_{i,j}, \forall j \neq i$  by the definition of  $\operatorname{Dom}(\mathbf{A})$ .
For a paper  $p$  in group  $i$ , consider the maximum total PQuality  $f$  of  $p$ . For some assignment  $\mathbf{x}$ , suppose  $\mathbf{x}$  assigns  $p$  to some reviewer  $r$  in group  $j \neq i$  with probability  $\tau$ . If we adjust  $\mathbf{x}$  so that the probability  $\tau$  is instead assigned to reviewers in group  $i$ , then the PQuality  $f$  will first decrease by  $f(\tau) \cdot A_{i,j} \leq \tau f'(0)A_{i,j}$  and then increase by at least  $\tau f'(1)A_{i,i}$ . As  $f'(0) < \operatorname{Dom}(\mathbf{A})f'(1)$ ,
$$
\tau f ^ {\prime} (1) A _ {i, i} - \tau f ^ {\prime} (0) A _ {i, j} > \tau f ^ {\prime} (1) \left(A _ {i, i} - \operatorname {D o m} (\mathbf {A}) A _ {i, j}\right) \geq 0.
$$
This shows that the adjustment increases the total PQuality  $f$  of  $p$ . Therefore, to maximize the total PQuality  $f$  of  $p$ ,  $p$  should only be assigned to reviewers in group  $i$ . Let  $p$  be assigned to the  $j$ -th reviewer in group  $i$  with probability  $\alpha_{j} \left( \sum_{j} \alpha_{j} = \ell_{p} \right)$ , then the total PQuality  $f$  of  $p$  equals
$$
\sum_ {j = 1} ^ {r _ {i}} f \left(\alpha_ {j}\right) \cdot A _ {i, i}. \tag {4}
$$
According to Jensen's inequality, only  $\alpha_{j} = \frac{\ell_{p}}{r_{i}},\forall j$  maximizes (4). This shows that, for every paper  $p$ , the corresponding row of  $\mathbf{x}^*$  is the unique assignment that maximizes the total PQuality  $f$  of  $p$ . And thus  $\mathbf{x}^*$  is the unique solution that maximizes the overall PQuality  $f$ .
Now we are ready to proceed to prove Theorem 2. Note that Claim B.2.2 shows that the only possible solution of PM is  $\mathbf{x}^*$ . For clearer presentation, let  $\mathbf{x}^{(\mathrm{PM})} = \mathbf{x}^*$ .
For Quality, Claim B.2.1 implies  $\forall \mathbf{x}^{(\mathrm{PLRA})}\in \mathrm{PLRA}(Q)$ ,  $\mathrm{Quality}(\mathbf{x}^{(\mathrm{PM})})\geq \mathrm{Quality}(\mathbf{x}^{(\mathrm{PLRA})})$ .
For Maxprob, compute that  $\mathrm{Maxprob}(\mathbf{x}^{(\mathrm{PM})}) = \max_i\{\ell_p / r_i\}$ . On the other hand, Corollary B.2.1 implies that  $\forall \mathbf{x}^{(\mathrm{PLRA})} \in \mathrm{PLRA}(Q)$ ,  $\mathrm{Maxprob}(\mathbf{x}^{(\mathrm{PLRA})}) \geq \max_i\{\ell_p / r_i\} = \mathrm{Maxprob}(\mathbf{x}^{(\mathrm{PM})})$ .
For Support, compute that  $\mathrm{Support}(\mathbf{x}^{(\mathrm{PM})}) = \sum_{i=1}^{k} p_i r_i$ . On the other hand, Corollary B.2.1 implies that  $\forall \mathbf{x}^{(\mathrm{PLRA})} \in \mathrm{PLRA}(Q)$ ,  $\mathrm{Support}(\mathbf{x}^{(\mathrm{PLRA})}) \leq \sum_{i=1}^{k} p_i r_i = \mathrm{Support}(\mathbf{x}^{(\mathrm{PM})})$ .
For the other metrics,  $\operatorname{AvgMaxp}$ , Entropy and L2Norm, consider a paper  $p$  in group  $i$ . By Corollary B.2.1, we know that PLRA only assigns  $p$  to reviewers in group  $i$ . Let it be assigned to the  $j$ -th reviewer in group  $i$  with probability  $\alpha_{j}$  ( $\sum_{j} \alpha_{j} = \ell_{p}$ ). Note that  $\max_{j} \{\alpha_{j}\}$  is minimized by  $\alpha_{j} = \frac{\ell_{p}}{r_{i}}, \forall j$ , which is the corresponding row in  $x^{(\mathrm{PM})}$  for each  $p$ . Meanwhile, according to Jensen's inequality,  $\sum_{j} \alpha_{j}^{2}$  is minimized by  $\alpha_{j} = \frac{\ell_{p}}{r_{i}}, \forall j$  and  $\sum_{j} \alpha_{j} \ln(1 / \alpha_{j})$  is maximized by  $\alpha_{j} = \frac{\ell_{p}}{r_{i}}, \forall j$ . So  $\forall x^{(\mathrm{PLRA})} \in \mathrm{PLRA}(Q)$ ,  $\operatorname{AvgMaxp}(x^{(\mathrm{PM})}) \leq \operatorname{AvgMaxp}(x^{(\mathrm{PLRA})})$ , L2Norm  $(x^{(\mathrm{PM})}) \leq L2\mathrm{Norm}(x^{(\mathrm{PLRA})})$  and Entropy  $(x^{(\mathrm{PM})}) \geq Entropy(x^{(\mathrm{PLRA})})$ . Moreover, as  $\{r_1, \ldots, r_k\}$  are not all the same, for some group  $i$ ,  $\frac{\ell_{p}}{r_{i}} < \max_{j} \left\{\frac{\ell_{p}}{r_{j}}\right\} \leq Q$ . For this group,  $\alpha_{j} = \frac{\ell_{p}}{r_{i}}, \forall j$  is not the only solution. Thus,  $\exists x^{(\mathrm{PLRA})} \in \mathrm{PLRA}(Q)$ ,  $\operatorname{AvgMaxp}(x^{(\mathrm{PM})}) < \operatorname{AvgMaxp}(x^{(\mathrm{PLRA})})$ , L2Norm  $(x^{(\mathrm{PM})}) < L2\mathrm{Norm}(x^{(\mathrm{PLRA})})$  and Entropy  $(x^{(\mathrm{PM})}) > Entropy(x^{(\mathrm{PLRA})})$ .
This concludes the proof of Theorem 2
B.3 Proof of Theorem 3