SlowGuess/ACL_2020_OCR huggingface dataset:
In order to prove that TBG satisfies the speed-biased criterion, consider two individuals ranked at consecutive positions  $k$  and  $k + 1$ ; if we swap the two individual, the change in TBG score is:
$$
\begin{array}{l} \Delta \mathrm {T B G} = (g _ {k + 1} - g _ {k}) D (T (k)) \\ + g _ {k} D (T (k) + t (k + 1)) \tag {16} \\ - g _ {k + 1} D (T (k) + t (k)) \\ \end{array}
$$
This leads to Lemma B.1-B.3:
Lemma B.1. Swapping a not-at-risk individual ranked at  $k$  with an at-risk individual ranked at  $k + 1$  always increases TBG.
Proof. Let  $g_{k} = 0$  and  $g_{k + 1} > 0$ . Equation 16 simplifies to
$$
\Delta \mathrm {T B G} = g _ {k + 1} (D (T (k)) - D (T (k) + t (k))) \tag {17}
$$
which is always positive because the decay function monotonically decreases, and each assessment of an individual requires at least  $T_{s}$  seconds.
Lemma B.2 (Risk-based Criterion). The optimal value of TBG under binary relevance is obtained only if all not-at-risk individuals are ranked below all at-risk individuals.
Proof. Let  $\pi$  be a ranking of individuals that yields the optimal value of TBG. Assume that in  $\pi$  there exist not-at-risk individuals ranked before at-risk individuals. Let the  $k$ -position be the lowest ranked not-at-risk individual that is at least in front of one at-risk individual, we can then apply Lemma B.1 to increase TBG. This leads to a contradiction.
Lemma B.3. Swapping an at-risk individual of longer assessment time ranked at  $k$  of with an at-risk individual of shorter assessment time ranked at  $k + n$ , where  $k + n$  is the closest at-risk individual ranked lower than  $k$ , always increases TBG.
Proof. Let  $g_{k} = g_{k + n} > 0$ , and  $\forall i \in \{i | k < i < k + n\}$ ,  $g_{i} = 0$ . We have
$$
\begin{array}{l} \Delta \mathrm {T B G} = g _ {k} (D (T (k + n) + t (k + n) - t (k)) \\ - D (T (k + n))) \tag {18} \\ \end{array}
$$
which is always positive because the decay function monotonically decreases, and  $t(k + n) < t(k)$  from the assumption that the individual at  $k + n$  has shorter assessment time.
Lemma B.3 naturally leads to a proof for the speed-biased property of TBG:
Proof for Theorem 3.1. Applying Lemma B.3, we know that swapping  $k$  and  $k + r$  leads to a positive gain between the two. Now, consider all
8135
at-risk individuals ranked between  $k$  and  $k + r$ :  $\forall u,$  s.t.  $k < u < k + r$ , the difference is:
$$
g _ {u} (D (T (u) + t (k + r) - t (k)) - D (T (u))) \tag {19}
$$
which is always greater than or equal to zero due to the fact that the decay function monotonically decrease, and  $t(k + r) < t(k)$ . Thus, the net difference is always larger than zero, thus satisfying the speed-biased criterion.
Finally, combing previous results, we can easily show:
Proof for Theorem 3.2. A direct consequence of Theorem 3.1 is that if the at-risk individuals are sorted by assessment time in ascending order, no swapping between any two individuals can increase TBG. This, combined with Lemma B.2, that all at-risk individuals are on top of not-at-risk individuals, leads to the necessary condition. Because any swapping within the not-at-risk individuals does not change TBG when no at-risk individuals are ranked lower, this implies that ranking according to Theorem 3.2 gives us a unique and optimal value, which satisfies the sufficient condition of Theorem 3.2.  $\square$
B.2 Hierarchical Time-Biased Gain
The assessment time of an individual ranked at  $k$ ,  $t(k)$ , is monotonic with  $E_{i}$ , thus showing minimal value of  $E_{i}$  suffices. Recall that  $E_{i}$  is calculated as:
$$
E _ {i} = T _ {\alpha} \sum_ {l = 1} ^ {L} \left(W _ {i, l} \prod_ {m = 1} ^ {l - 1} \left(1 - R _ {i, m}\right)\right) + T _ {\beta} (2 0)
$$
Consider, again, swapping a document at rank  $l$  with a document at rank  $l + 1$  belonging to the same individual  $i$ . The change in  $E_{i}$  is:
$$
\Delta E _ {i} = \kappa_ {i, l} \left(W _ {i, l + 1} R _ {i, l} - W _ {i, l} R _ {i, l + 1}\right) \tag {21}
$$
where  $\kappa_{i,l} = T_{\alpha}\prod_{j = 1}^{l - 1}(1 - R_{i,j})\geq 0$  is a fixed term that is not affected by the swap.
Equation 21 also points to an important observation:
Lemma B.4. If  $W_{i,l+1}R_{i,l} - W_{i,l}R_{i,l+1} < 0$  and  $R_{i,j} < 1$  for all  $j < l$ , then swapping document  $l$  with document  $l + 1$  will decrease  $E_{i}$ .
Proof. This follows directly from Equation 21.  $\square$
Lemma B.5. If  $R_{i,j} < 1$  for all  $j$ , then minimum individual assessment time is obtained if and only if the documents are sorted in descending order by
$$
\frac {R _ {i , l}}{W _ {i , l}}. \tag {22}
$$
Proof. Let  $\tau$  be a document ranking that yields the minimum individual assessment time, and for the sake of contradiction, not a ranking that can be obtained by ranking according to  $\frac{R_{i,l}}{W_{i,l}}$ . We can, thus, find two neighboring documents, without loss of generality,  $l$  and  $l + 1$ , such that:
$$
\frac {R _ {i , l}}{W _ {i , l}} <   \frac {R _ {i , l + 1}}{W _ {i , l + 1}} \tag {23}
$$
this leads to:
$$
R _ {i, l} W _ {i, l + 1} - R _ {i, l + 1} W _ {i, l} <   0 \tag {24}
$$
since all  $W > 0$ . Lemma B.4 together with the prerequisite that  $R_{i,j} < 1$  for all  $j$  then suggest that swapping the two leads to a decrease of  $E_{i}$ . This contradicts with the assumption that  $\tau$  is an optimal ranking. This proves that to achieve minimum individual assessment time, it is necessary to sort by  $\frac{R_{i,l}}{W_{i,l}}$ . The sufficient condition follows by the fact that swapping tied documents does not lead to change in  $E_{i}$ , as shown in Equation 21
Proof for Theorem 3.3. Let  $\tau$  be a document ranking according to  $\frac{R_{i,l}}{W_{i,l}}$ . Let  $m$  be the document such that  $R_{i,m} = 1$  and is ranked closer to the top then any other document with  $R_{i,:} = 1$  (i.e. with the shortest  $W_{i,:}$ ). Now, consider using  $m$  to cut the documents into two partitions: the first partition of documents are ones ranked before  $m$ . Applying Lemma B.5, this partition of documents are already in optimal sorted order, since there's no  $R_{i,:} = 1$ . The second partition, documents ranked lower than  $m$ , the ranking simply does not matter, as Equation 20 shows, the  $(1 - R_{i,m})$  term will make everything zero afterwards.
Now, let's consider moving a document from the second partition to the first partition. Since any documents in the second partition has a  $\frac{R_{i,j}}{W_{i,j}}$  that is smaller than any documents in the first partition, after you move the document, the optimal ranking for the first partition will put the document at the bottom, right next to  $m$ . And since  $\frac{R_{i,m}}{W_{i,m}} \geq \frac{R_{i,j}}{W_{i,j}}$  due to the original ordering, we can apply Lemma B.4, which can swap the document back below  $m$ . Next,
8136
consider moving the lowest ranked document of the first partition (the one ranked at  $m - 1$ ) to the second partition. This will always increase  $E_{i}$ , as shown from Lemma B.4. Moving any other document in the first partition will also increase  $E_{i}$  as least as much as before, since the process is equivalent to swapping with (and thus potentially increase  $E_{i}$ ) any intermediate documents in between.
Combine these two together, we show that  $E_{i}$  is at a minimum value when sorted in descending order according to  $\frac{R_{i,l}}{W_{i,l}}$ .
B.3 Relationship between ERR and hTBG
Here we show the derivation from the cascading user model in ERR to the individual assessment time estimation  $(E_{i})$  in hTBG. ERR assumes a stopping probability (written in hTBG terms):
$$
P (\text {s t o p} 1) = R _ {i, l} \prod_ {j = 1} ^ {l - 1} \left(1 - R _ {i, j}\right) \tag {25}
$$
The expected words read, can then be calculated as:
$$
\begin{array}{l} \sum_ {l = 1} ^ {L} \left(P (\text {s t o p a t} l) \sum_ {d = 1} ^ {l} W _ {i, l}\right) \\ = \sum_ {l = 1} ^ {L} \left(R _ {i, l} \prod_ {j = 1} ^ {l - 1} \left(1 - R _ {i, j}\right) \left(\sum_ {d = 1} ^ {l} W _ {i, l}\right)\right) \tag {26} \\ \end{array}
$$
This can be rearranged to the formula we used in hTBG:
$$
\sum_ {l = 1} ^ {L} \left(W _ {i, l} \prod_ {m = 1} ^ {l - 1} \left(1 - R _ {i, m}\right)\right) \tag {27}
$$
by letting  $R_{i,L} = 1$  (the user has to stop reading at the last document). To show this, observe that  $W_{i,1}$  appears in all  $L$  terms of the summation, thus the coefficient for  $W_{i,1}$  is simply  $\sum_{l=1}^{L}(R_{i,l}\prod_{j=1}^{l-1}(1 - R_{i,j})) = 1$ , from both simple manipulation and the fact that we are summing over probability. Similarly,  $W_{i,2}$  appears in all  $L$  terms except with  $l = 1$ , thus  $(1 - R_{i,1})$ . For  $W_{i,3}$  it is  $(1 - R_{i,1}) - R_{i,2}(1 - R_{i,1}) = \prod_{j=1}^{2}(1 - R_{i,j})$ . The rest follows.
C Appendix: Training Details
All models are built using AllenNLP (Gardner et al., 2018). Tokenization and sentence splitting are done using spaCy (Honnibal and Johnson,
The CROWDSOURCE dataset is split into a training set  $(80\%)$  and a validation set  $(20\%)$  during model development. We did not test on the Expert dataset until all parameters of the models were fixed. Cross validation on the training set is used for hyperparameter tuning. For 3HAN, we used ADAM with learning rate 0.003, trained for 100 epochs with early stopping on the validation dataset, with patience set to 30. For 3HAN_Av, the same hyperparameters are used. For LR, we used SGD with learning rate 0.003, trained for 100 epochs with early stopping on the validation dataset, with patience set to 30.
Both 3HAN and 3HAN_Av's Seq2Vec layers use bi-directional GRU with attention. The word-to-sentence layer has input dimension of 200, hidden dimension of 50, and output dimension of 100, since the GRU is bi-directional. The sentence-to-document and document-to-individual layer, similarly, has input dimension of 100, hidden dimension of 50, and output dimension of 100. Hyperparameters were selected using cross validation on the training set split of the CROWDSOURCE dataset.
8137