SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Can (Sam) Chen $^{1,2*}$ , Christopher Beckham $^{2,3}$ , Zixuan Liu $^{5}$ , Xue Liu $^{1,2}$ , Christopher Pal $^{2,3,4}$
<sup>1</sup>McGill University, <sup>2</sup>MILA - Quebec AI Institute,
$^{3}$ Polytechnique Montreal,  $^{4}$ Canada CIFAR AI Chair,  $^{5}$ University of Washington
We study offline model-based optimization to maximize a black-box objective function with a static dataset of designs and scores. These designs encompass a variety of domains, including materials, robots and DNA sequences. A common approach trains a proxy on the static dataset to approximate the black-box objective function and performs gradient ascent to obtain new designs. However, this often results in poor designs due to the proxy inaccuracies for out-of-distribution designs. Recent studies indicate that: (a) gradient ascent with a mean ensemble of proxies generally outperforms simple gradient ascent, and (b) a trained proxy provides weak ranking supervision signals for design selection. Motivated by (a) and (b), we propose parallel-mentoring as an effective and novel method that facilitates mentoring among parallel proxies, creating a more robust ensemble to mitigate the out-of-distribution issue. We focus on the three-proxy case and our method consists of two modules. The first module, voting-based pairwise supervision, operates on three parallel proxies and captures their ranking supervision signals as pairwise comparison labels. These labels are combined through majority voting to generate consensus labels, which incorporate ranking supervision signals from all proxies and enable mutual mentoring. However, label noise arises due to possible incorrect consensus. To alleviate this, we introduce an adaptive soft-labeling module with soft-labels initialized as consensus labels. Based on bi-level optimization, this module fine-tunes proxies in the inner level and learns more accurate labels in the outer level to adaptively mentor proxies, resulting in a more robust ensemble. Experiments validate the effectiveness of our method. Our code is available here.
Designing new objects or entities to optimize specific properties is a widespread challenge, encompassing various domains such as materials, robots, and DNA sequences [1]. Traditional approaches often involve interacting with a black-box function to propose new designs, but this can be expensive or even dangerous in some cases [2-6]. In response, recent work [1] has focused on a more realistic setting known as offline model-based optimization (MBO). In this setting, the objective is to maximize a black-box function using only a static (offline) dataset of designs and scores.
A prevalent approach to addressing the problem is to train a deep neural network (DNN) model parameterized as  $f_{\theta}(\cdot)$ , on the static dataset, with the trained DNN serving as a proxy. The proxy allows for gradient ascent on existing designs, generating improved designs by leveraging the gradient information provided by the DNN model. However, this method encounters an issue with the trained proxy being susceptible to out-of-distribution problems. Specifically, the proxy produces inaccurate predictions when applied to data points that deviate significantly from the training distribution.
*Correspondence to can.chen@mila.quebec.
Recent studies have observed that (a) employing a mean ensemble of trained proxies for gradient ascent in offline MBO generally leads to superior designs compared to using a single proxy [7].
This improvement stems from the ability of the ensemble to provide more robust predictions compared to a single proxy [8-11]. Recent work has also found that (b) a trained proxy offers weak (valuable, albeit potentially unreliable) ranking supervision signals for design selection in various offline MBO contexts, such as evolutionary algorithms [12], reinforcement learning [13], and generative modeling [14]. These signals, focusing on the relative order of designs over absolute scores, are more resilient to noise and inaccuracies. By exchanging these signals among proxies in the ensemble, we can potentially enhance its robustness. As shown in Figure 1, we have three parallel proxies  $f_{\theta}^{A}(\cdot), f_{\theta}^{B}(\cdot)$  and  $f_{\theta}^{C}(\cdot)$ . For two designs
Figure 1: Motivation illustration.
$\pmb{x}_1^n$  and  $\pmb{x}_2^n$  within the neighborhood of the current optimization point, proxies  $f_{\theta}^{A}(\cdot)$  and  $f_{\theta}^{B}(\cdot)$  agree that the score of  $\pmb{x}_1^n$  is larger than that of  $\pmb{x}_2^n$ , while proxy  $f_{\theta}^{C}(\cdot)$  disagrees. Based on the majority voting principle, proxies  $f_{\theta}^{A}(\cdot)$  and  $f_{\theta}^{B}(\cdot)$  provide a more reliable ranking, and their voted ranking signal  $f^{V}(\pmb{x}_{1}^{n}) > f^{V}(\pmb{x}_{2}^{n})$  could mentor the proxy  $f_{\theta}^{C}(\cdot)$ , thus enhancing its performance.
To this end, we propose an effective and novel method called parallel-mentoring that facilitates mentoring among parallel proxies to train a more robust ensemble against the out-of-distribution issue. This paper primarily focuses on the three-proxy case, referred to as tri-mentoring, but we also examine the situation with more proxies in Appendix A.1. As depicted in Figure 2, tri-mentoring consists of two modules. Module 1, voting-based pairwise supervision (shown in Figure 2(a)), operates on three parallel proxies  $f_{\theta}^{A}(\cdot)$ ,  $f_{\theta}^{B}(\cdot)$ , and  $f_{\theta}^{C}(\cdot)$  and utilizes their mean for the final prediction. To ensure consistency with the ranking information employed in design selection, this module adopts a pairwise approach to represent the ranking signals of each proxy. Specifically, as illustrated in Figure 2(a), this module generates samples (e.g.  $x_{1}^{n}$ ,  $x_{2}^{n}$  and  $x_{3}^{n}$ ) in the neighborhood of the current point  $x$  and computes pairwise comparison labels  $\hat{y}^{A}$  for all sample pairs, serving as ranking supervision signals for the proxy  $f_{\theta}^{A}(\cdot)$ . The label  $\hat{y}_{ij}^{A}$  is defined as 1 if  $f_{\theta}^{A}(x_{i}) > f_{\theta}^{A}(x_{j})$  and 0 otherwise, and similar signals are derived for proxies  $f_{\theta}^{B}(\cdot)$  and  $f_{\theta}^{C}(\cdot)$ . These labels  $\hat{y}^{A}, \hat{y}^{B}$  and  $\hat{y}^{C}$  are combined via majority voting to create consensus labels  $\hat{y}^{V}$  which are more reliable and thus can be used for mentoring proxies. The voted ranking signal  $f^{V}(x_{1}^{n}) > f^{V}(x_{2}^{n})$  in Figure 1 corresponds to the pairwise consensus label  $\hat{y}_{12}^{V} = 1$  in Figure 2(a), and both can mentor the proxy  $f_{\theta}^{C}(\cdot)$ .
Figure 2: Illustration of tri-mentoring.
Module 2, adaptive soft-labeling (shown in Figure 2(b)), mitigates the issue of label noise that may arise, since the voting consensus may not always be correct. To this end, this module initializes the consensus labels  $\hat{\pmb{y}}^V$  from the first module as soft-labels  $\hat{\pmb{y}}^S$ . It then aims to learn more accurate soft-labels to better represent the ranking supervision signals by leveraging the knowledge from the static dataset. Specifically, assuming accurate soft-labels, one of the proxies, either  $f_{\theta}^{A}(\cdot)$ ,  $f_{\theta}^{B}(\cdot)$  or  $f_{\theta}^{C}(\cdot)$ , fine-tuned using them, is expected to perform well on the static dataset, as both soft-labels
(pairwise perspective) and the static dataset (pointwise perspective) describe the same ground-truth and share underlying similarities. This formulation leads to a bi-level optimization framework with an inner fine-tuning level and an outer soft-labeling level as shown in Figure 2(b). The inner level fine-tunes the proxy with soft-labels, which establishes the connection between them. The outer level optimizes soft-labels to be more accurate by minimizing the loss of the static dataset via the inner-level connection. The optimized labels are further fed back to the first module to adaptively mentor the proxy, ultimately yielding a more robust ensemble. Experiments on design-bench validate the effectiveness of our method.
- We propose parallel-mentoring for offline MBO, effectively utilizing weak ranking supervision signals among proxies, with a particular focus on the three-proxy case as tri-mentoring.
- Our method consists of two modules: voting-based pairwise supervision and adaptive soft-labeling. The first module generates pairwise consensus labels via majority voting to mentor the proxies.
- To mitigate label noise in consensus labels, the second module proposes a bi-level optimization framework to adaptively fine-tune proxies and soft-labels, resulting in a more robust ensemble.
2 Preliminaries: Gradient Ascent on Offline Model-based Optimization
Offline model-based optimization (MBO) aims to find the optimal design  $\boldsymbol{x}^*$  that maximizes the black-box objective function  $f(\cdot)$ :
$$
\boldsymbol {x} ^ {*} = \underset {\boldsymbol {x}} {\arg \max } f (\boldsymbol {x}), \tag {1}
$$
To achieve this, a static dataset  $\mathcal{D} = \{(x_i,y_i)\}_{i = 1}^N$  with  $N$  points is available, where  $x_{i}$  represents a design and  $y_{i}$  is its corresponding score.
A common approach for solving this optimization problem is to fit a deep neural network (DNN) model  $f_{\theta}(\cdot)$  with parameters  $\theta$  to the static dataset in a supervised manner. The optimal parameters  $\theta^{*}$  can be obtained by minimizing the mean squared error between the predictions and the true scores:
The trained DNN model  $f_{\theta^{*}}(\cdot)$  acts as a proxy to optimize the design using gradient ascent steps:
$$
\boldsymbol {x} _ {t + 1} = \boldsymbol {x} _ {t} + \eta \nabla_ {\boldsymbol {x}} f _ {\theta} (\boldsymbol {x}) | _ {\boldsymbol {x} = \boldsymbol {x} _ {t}}, \quad \text {f o r} t \in [ 0, T - 1 ], \tag {3}
$$
where  $T$  is the number of steps and  $\eta$  represents the learning rate.  $\pmb{x}_T$  serves as the final design candidate. However, this method faces a challenge with the proxy being vulnerable to out-of-distribution designs. When handling designs that substantially differ from the training distribution, the proxy yields inaccurate predictions.
In this section, we introduce parallel-mentoring, focusing on the three-proxy scenario, also known as tri-mentoring. The method can be easily extended to incorporate more proxies, as discussed in Appendix A.1. Tri-mentoring consists of two modules. The first module, voting-based pairwise supervision in Section 3.1, manages three proxies parallelly and generates consensus labels via majority voting to mentor proxies. To mitigate label noise, we introduce a second module, adaptive soft-labeling in Section 3.2. This module adaptively fine-tunes proxies and soft-labels using bi-level optimization, improving ensemble robustness. The overall algorithm is shown in Algorithm 1.
3.1 Voting-based Pairwise Supervision
We train three parallel proxies  $f_{\pmb{\theta}}^{A}(\cdot), f_{\pmb{\theta}}^{B}(\cdot)$  and  $f_{\pmb{\theta}}^{C}(\cdot)$  on the static dataset with different initializations and utilize their mean as the final prediction as suggested in [1, 15]:
$$
f _ {\boldsymbol {\theta}} (\cdot) = \frac {1}{3} \left(f _ {\boldsymbol {\theta}} ^ {A} (\cdot) + f _ {\boldsymbol {\theta}} ^ {B} (\cdot) + f _ {\boldsymbol {\theta}} ^ {C} (\cdot)\right). \tag {4}
$$
We then apply gradient ascent with  $f_{\theta}(\cdot)$  on existing designs to generate improved designs as per Eq.(3). Although the mean ensemble generally results in superior designs compared to a single proxy
Algorithm 1 Tri-mentoring for Offline Model-based Optimization
Input: The static dataset  $\mathcal{D}$  , the number of iterations  $T$  , the optimizer  $OPT(\cdot)$  Output: The high-scoring design  $\pmb{x}_h^*$    
1 Initialize  $\pmb{x_0}$  as the design with the highest score in  $\mathcal{D}$    
2 Train proxies  $f_{\theta}^{A}(\cdot),f_{\theta}^{B}(\cdot)$  and  $f_{\theta}^{C}(\cdot)$  on  $\mathcal{D}$  with different initializations.   
3 for  $t\gets 0$  to  $T - 1$  do  $\triangleright$  Voting-based pairwise supervision.   
4 Sample  $K$  neighborhood points at  $\pmb {x}_t$  as  $S(x_{t})$    
5 Compute pairwise comparison labels  $\hat{y}^A,\hat{y}^B$  and  $\hat{y}^C$  for three proxies on  $S(x_{t})$    
6 Derive consensus labels:  $\hat{y}^V =$  majority_voting  $(\hat{y}^A,\hat{y}^B,\hat{y}^C)$  .  $\triangleright$  Adaptive soft-labeling.   
7 for proxy in  $[f_{\theta}^{A}(\cdot),f_{\theta}^{B}(\cdot),f_{\theta}^{C}(\cdot)]$  do   
8 Initialize soft-labels as consensus labels:  $\hat{y}^S = \hat{y}^V$    
9 Inner level: fine-tune the proxy with Eq.(8).   
10 Outer level: learn more accurate soft-labels  $\hat{y}^S$  with Eq.(9).   
11 Mentor proxy using the optimized soft-labels  $\hat{y}^S$  with Eq.(8).  $\triangleright$  Gradient ascent with a mean ensemble.   
12 Form a more robust ensemble as  $f_{\theta}(\pmb {x}) = \frac{1}{3} (f_{\theta}^{A}(\pmb {x}) + f_{\theta}^{B}(\pmb {x}) + f_{\theta}^{C}(\pmb {x}))$    
13 Gradient ascent:  $x_{t + 1} = x_t + \eta OPT(\nabla_xf_\theta (x_t))$    
14 Return  $\pmb{x}_h^* = \pmb{x}_T$
[7] due to the ensemble robustness [8, 9], this approach does not fully exploit the potential of weak (valuable, albeit potentially unreliable) ranking supervision signals within every proxy. Emphasizing the relative order of designs rather than their absolute scores, these signals are more resilient to noise and inaccuracies. These ranking signals are commonly used in evolutionary algorithms [12], reinforcement learning [13], and generative modeling [14] to select designs and could further improve the ensemble robustness. We extract the ranking supervision signals from individual proxies in the form of pairwise comparison labels, and then combine these labels via majority voting to generate consensus labels to mentor proxies. We provide a detailed explanation of this module below, with its implementation shown in Algorithm 1 from Line 4 to Line 6.
Pairwise comparison label. We adopt a pairwise approach to represent the ranking supervision signals for every proxy, focusing on relative order to align with the ranking information used in design selection. We sample  $K$  points at the neighborhood of the optimization point  $\pmb{x}_t$  as  $S(\pmb{x}_t) = \{\pmb{x}_1^n,\dots,\pmb{x}_K^n\} \sim \mathcal{N}(\pmb{x}_t,\delta^2)$  where  $\mathcal{N}(\pmb{x}_t,\delta^2)$  represents a Gaussian distribution centered at  $\pmb{x}_t$  with variance  $\delta^2$ . For each sample pair  $(\pmb{x}_i^n,\pmb{x}_j^n)$  and a proxy (e.g.,  $f_{\theta}^{A}(\cdot)$ ), we define the pairwise comparison label  $\pmb{y}_{ij}^{A} = \mathbf{1}(f_{\theta}^{A}(\pmb{x}_{i}^{n}) > f_{\theta}^{A}(\pmb{x}_{j}^{n}))$ , where  $\mathbf{1}$  is the indicator function. The labels  $\hat{\pmb{y}}^A$  from all sample pairs serve as the ranking supervision signals for the proxy  $f_{\theta}^{A}(\cdot)$ . We repeat this process for all proxies, generating signals  $\hat{\pmb{y}}^B$  and  $\hat{\pmb{y}}^C$  for proxies  $f_{\theta}^{B}(\cdot)$  and  $f_{\theta}^{C}(\cdot)$  respectively.
Majority voting. Given these pairwise comparison labels  $\hat{\pmb{y}}^A$ ,  $\hat{\pmb{y}}^B$  and  $\hat{\pmb{y}}^C$ , we derive the pairwise consensus labels  $\hat{\pmb{y}}^V$  via an element-wise majority voting:
$$
\hat {\boldsymbol {y}} _ {i j} ^ {V} = \text {m a j o r i t y} _ {-} \text {v o t i n g} \left(\hat {\boldsymbol {y}} _ {i j} ^ {A}, \hat {\boldsymbol {y}} _ {i j} ^ {B}, \hat {\boldsymbol {y}} _ {i j} ^ {C}\right), \tag {5}
$$
where  $i$  and  $j$  are the indexes of the neighborhood samples. As consensus labels are generally more reliable, they can be employed for mentoring the proxies to promote the exchange of ranking supervision signals. Specifically, we can fine-tune the proxy  $f_{\theta}^{A}(\cdot)$  using the binary cross-entropy loss, where  $\sigma(f_{\theta}^{A}(\pmb{x}_{i}^{n}) - f_{\theta}^{A}(\pmb{x}_{j}^{n}))$  represents the predicted probability that  $f_{\theta}^{A}(\pmb{x}_{i}^{n}) > f_{\theta}^{A}(\pmb{x}_{j}^{n})$ , as also used in the ChatGPT reward model training [16-18]. The loss function can be computed as:
$$
\mathcal {L} ^ {A} (\boldsymbol {\theta}) = - \frac {1}{C _ {K} ^ {2}} \sum_ {1 \leq i <   j \leq K} \hat {\boldsymbol {y}} _ {i j} ^ {V} \log \left[ \sigma \left(f _ {\boldsymbol {\theta}} ^ {A} \left(\boldsymbol {x} _ {i} ^ {n}\right) - f _ {\boldsymbol {\theta}} ^ {A} \left(\boldsymbol {x} _ {j} ^ {n}\right)\right) \right] + \left(1 - \hat {\boldsymbol {y}} _ {i j} ^ {V}\right) \log \left[ \sigma \left(f _ {\boldsymbol {\theta}} ^ {A} \left(\boldsymbol {x} _ {j} ^ {n}\right) - f _ {\boldsymbol {\theta}} ^ {A} \left(\boldsymbol {x} _ {i} ^ {n}\right)\right) \right], \tag {6}
$$
where  $C_K^2 = \frac{K(K - 1)}{2}$  denotes the number of the sample pairs. This procedure is also applied to proxies  $f_{\theta}^{B}(\cdot)$  and  $f_{\theta}^{C}(\cdot)$ . While our approach encourages alignment with the consensus, it does not aim to make proxies identical. Each proxy maintains its unique learning trajectory, thereby preserving the diversity among the proxies.
3.2 Adaptive Soft-labeling
The consensus labels  $\hat{\pmb{y}}^V$  may contain noise since the majority voting consensus can be incorrect. To mitigate this issue, we propose an adaptive soft-labeling module. This module initializes soft-labels as consensus labels, and employs a bi-level optimization framework for adaptive mentoring, where the inner level fine-tunes the proxies and the outer level refines the soft-labels. We provide a detailed description of this module below; its implementation is outlined in Algorithm 1, Line 7- Line 11.
Fine-tuning. We initialize the soft-labels as the consensus labels:  $\hat{\pmb{y}}^S = \hat{\pmb{y}}^V$ , serving as an effective starting point. Utilizing the soft-labels  $\hat{\pmb{y}}^S$ , we can perform fine-tuning on the proxy  $f_{\pmb{\theta}}^{A}(\cdot)$  against the binary cross-entropy loss following Eq.(6),
$$
\mathcal {L} ^ {A} (\boldsymbol {\theta}, \boldsymbol {y} ^ {S}) = - \frac {1}{C _ {K} ^ {2}} \sum_ {1 \leq i <   j \leq K} \hat {\boldsymbol {y}} _ {i j} ^ {S} \log \left[ \sigma \left(f _ {\boldsymbol {\theta}} ^ {A} \left(\boldsymbol {x} _ {i}\right) - f _ {\boldsymbol {\theta}} ^ {A} \left(\boldsymbol {x} _ {j}\right)\right) \right] + \left(1 - \hat {\boldsymbol {y}} _ {i j} ^ {S}\right) \log \left[ \sigma \left(f _ {\boldsymbol {\theta}} ^ {A} \left(\boldsymbol {x} _ {j}\right) - f _ {\boldsymbol {\theta}} ^ {A} \left(\boldsymbol {x} _ {i}\right)\right) \right]. \tag {7}
$$
In contrast to Eq.(6), we express the loss  $\mathcal{L}^A (\theta ,\hat{\pmb{y}}^S)$  as a function of both the proxy parameters  $\pmb{\theta}$  and the soft-labels  $\hat{\pmb{y}}^S$  as the accurate soft-labels  $\hat{\pmb{y}}^S$  have yet to be determined. One-step gradient descent enables fine-tuning, resulting in the following relationship:
$$
\boldsymbol {\theta} \left(\hat {\boldsymbol {y}} ^ {S}\right) = \boldsymbol {\theta} - \gamma \frac {\partial \mathcal {L} ^ {A} \left(\boldsymbol {\theta} , \hat {\boldsymbol {y}} ^ {S}\right)}{\partial \boldsymbol {\theta} ^ {\top}}, \tag {8}
$$
where  $\gamma$  denotes the fine-tuning learning rate.
Soft-labeling. Assuming the soft-labels are accurate, the fine-tuned proxy  $f_{\theta(\hat{\boldsymbol{y}}^S)}^A(\cdot)$  is expected to perform well on the static dataset. This is due to the fact that, despite having different data distributions, the soft-labels and the static dataset share underlying similarities, as they both represent the same ground-truth from pairwise and pointwise perspectives, respectively. This leads to a bi-level optimization framework with the fine-tuning mentioned above as the inner level and the soft-labeling here as the outer level. In particular, we enhance the accuracy of soft-labels  $\hat{\boldsymbol{y}}^S$  by minimizing the mean squared error on the static dataset  $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N$ ,
$$
\hat {\boldsymbol {y}} ^ {S ^ {\prime}} = \hat {\boldsymbol {y}} ^ {S} - \frac {\lambda}{N} \frac {\partial \sum_ {i = 1} ^ {N} \left(f _ {\boldsymbol {\theta} (\hat {\boldsymbol {y}}) ^ {s}} ^ {A} \left(\boldsymbol {x} _ {i}\right) - y _ {i}\right) ^ {2}}{\partial \left(\hat {\boldsymbol {y}} ^ {S}\right) ^ {\top}}, \tag {9}
$$
where  $\lambda$  represents the soft-labeling learning rate. The nested optimization problem can be easily solved by higher, a library for higher-order optimization [19]. Once optimized, the labels are fed back to the first module, adaptively mentoring the proxy  $f_{\theta}^{A}(\cdot)$  according to Eq.(8). The same procedure can be employed for proxies  $f_{\theta}^{B}(\cdot)$  and  $f_{\theta}^{C}(\cdot)$ , ultimately resulting in a more robust ensemble. We further clarify the novelty of this module in Appendix A.2.
We conduct extensive experiments on design-bench to investigate the effectiveness and robustness of the proposed method. In Section 4.4, we benchmark our approach against several well-established baselines. In Section 4.5, we verify the effectiveness of two modules: voting-based pairwise supervision and adaptive soft-labeling, as well as other contributing factors. In Section 4.6, we investigate the sensitivity of our method to hyperparameter changes. While our primary focus is the tri-mentoring with 3 proxies, we additionally explore other parallel-mentoring implementations utilizing 5, 7, 9, and 11 proxies in Appendix A.1.
4.1 Task Overview
We adopt the design-bench which comprises both continuous and discrete tasks. Below, we outline the dataset details and evaluation protocols.
Dataset. We conduct experiments on four continuous tasks: (1) Superconductor (SuperC) [2]: discover an 86-D superconductor to maximize critical temperature with 17010 designs. (2) Ant Morphology (Ant) [20]: identify a 60-D ant morphology to crawl quickly with 10004 designs. (3) D'Kitty Morphology (D'Kitty) [21]: determine a 56-D D'Kitty morphology to crawl quickly with 10004 designs. (4) Hopper Controller (Hopper) [1]: find a neural network policy with 5126 weights to maximize return with 3200 designs. Besides, we perform experiments on four discrete tasks: (1) TF Bind 8 (TFB8) [5]: design a length 8 DNA sequence to maximize binding activity score with 32896 designs. (2) TF Bind 10 (TFB10) [5]: find a length 10 DNA sequence to maximize binding
activity score with 50000 designs. (3) NAS [1]: find a 64-D NN with 5 categories per dimension to maximize the performance on CIFAR10 with 1771 designs.
Evaluation. We use the oracle evaluation of design-bench to evaluate a certain design and the details of the oracle functions are reported in Design-Bench Benchmark Tasks in [1]. Following [7], we select the top  $\mathrm{N} = 128$  candidates for each method and report the  $100^{th}$  percentile (maximum) normalized ground truth score. The score, denoted as  $y_{n}$  is computed as  $\frac{y - y_{min}}{y_{max} - y_{min}}$  where  $y$  is the design score, and  $y_{min}$  and  $y_{max}$  are the lowest and highest scores in the complete unobserved dataset, respectively. In addition, we provide the  $50^{th}$  percentile (median) normalized ground truth scores used in the prior work in Appendix A.3. We also provide the mean and median ranks of all comparison methods to better assess the overall performance.
4.2 Comparisons with Other Methods