SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ye Yuan $^{1*}$ , Can (Sam) Chen $^{1,2*†}$ , Zixuan Liu $^{3}$ , Willie Neiswanger $^{4}$ , Xue Liu $^{1}$
<sup>1</sup> McGill University, <sup>2</sup> MILA - Quebec AI Institute,
<sup>3</sup> University of Washington, <sup>4</sup> Stanford University
ye.yuan3@mail.mcgill.ca, can.chen@mila.quebec,
zucksliu@cs.washington.edu, neiswanger@cs.stanford.edu,
xueliu@cs.mcgill.ca
Offline model-based optimization aims to find a design that maximizes a property of interest using only an offline dataset, with applications in robot, protein, and molecule design, among others. A prevalent approach is gradient ascent, where a proxy model is trained on the offline dataset and then used to optimize the design. This method suffers from an out-of-distribution issue, where the proxy is not accurate for unseen designs. To mitigate this issue, we explore using a pseudolabeler to generate valuable data for fine-tuning the proxy. Specifically, we propose Importance-aware Co-Teaching for Offline Model-based Optimization (ICT). This method maintains three symmetric proxies with their mean ensemble as the final proxy, and comprises two steps. The first step is pseudo-label-driven co-teaching. In this step, one proxy is iteratively selected as the pseudo-labeler for designs near the current optimization point, generating pseudo-labeled data. Subsequently, a co-teaching process identifies small-loss samples as valuable data and exchanges them between the other two proxies for fine-tuning, promoting knowledge transfer. This procedure is repeated three times, with a different proxy chosen as the pseudolabeler each time, ultimately enhancing the ensemble performance. To further improve accuracy of pseudo-labels, we perform a secondary step of meta-learning-based sample reweighting, which assigns importance weights to samples in the pseudo-labeled dataset and updates them via meta-learning. ICT achieves state-of-the-art results across multiple design-bench tasks, achieving the best mean rank of 3.1 and median rank of 2, among 15 methods. Our source code can be found here.
A primary goal in many domains is to design or create new objects with desired properties [1]. Examples include the design of robot morphologies [2], protein design, and molecule design [3, 4]. Numerous studies obtain new designs by iteratively querying an unknown objective function that maps a design to its corresponding property score. However, in real-world scenarios, evaluating the objective function can be expensive or risky [3-7]. As a result, it is often more practical to assume access only to an offline dataset of designs and their property scores. This type of problem is referred to as offline model-based optimization (MBO) [1]. The goal of MBO is to find a design that maximizes the unknown objective function using solely the offline dataset.
*Equal contribution with random order.
Gradient ascent is a common approach to address the offline MBO problem. For example, as illustrated in Figure 2 (a), the offline dataset may consist of three robot size and robot speed pairs  $p_{1,2,3}$ . A simple DNN model, referred to as the vanilla proxy and represented as  $f_{\theta}(\cdot)$ , is trained to fit the offline dataset as an approximation to the unknown objective function. Gradient ascent is subsequently applied to existing designs with respect to the vanilla proxy  $f_{\theta}(\cdot)$ , aiming to generate a new design with a higher score. However, the gradient ascent method suffers from an out-of-distribution issue, where the vanilla proxy cannot accurately estimate data outside of the training distribution, leading to a significant gap between the vanilla proxy and the ground-truth function, as shown in Figure 2 (a). As a consequence, the scores of new designs obtained via gradient ascent can be erroneously high [8, 9].
To mitigate the out-of-distribution issue, recent studies have suggested applying regularization techniques to either the proxy itself [8-10] or the design under consideration [11, 12]. These methods improve the proxy's robustness and generalization ability. However, a yet unexplored approach in this domain is using a pseudo-labeler to assign pseudo-labels to designs near the current point. Fine-tuning the proxy on this pseudo-labeled dataset can lead to improvement, provided that we can identify the valuable portion of the pseudo-labeled dataset.
Inspired by this, we propose Importance-aware Co-Teaching for Offline Model-based Optimization (ICT). This approach maintains three symmetric proxies, and their mean ensemble acts as the final proxy. ICT consists of two main steps with the first step being pseudo-label-driven co-teaching as illustrated in Figure 1. During this step, one proxy is iteratively selected as the pseudo-labeler, followed by a co-teaching process [13] that facilitates the exchange of valuable data between the other two proxies for fine-tuning. As depicted in Figure 1, there are three symmetric proxies,  $f_{\theta_1}(\cdot)$ ,  $f_{\theta_2}(\cdot)$ , and  $f_{\theta_3}(\cdot)$ . The entire learning cycle (the larger triangle) can be divided into three symmetric parts (sub-triangles), with one proxy chosen to be the
Figure 1: Pseudo-label-driven co-teaching.
pseudo-labeler in turn. Taking the top triangle as an example, we select  $f_{\theta_1}(\cdot)$  as the pseudo-labeler to generate pseudo labels for a set of points in the neighborhood of the current optimization point  $x_t$ . The other two proxies,  $f_{\theta_2}(\cdot)$  and  $f_{\theta_3}(\cdot)$ , then receive the pseudo-labeled dataset. They compute the sample loss for each entry in the dataset and exchange small-loss samples between them for fine-tuning. This co-teaching process encourages knowledge transfer between the two proxies, as small losses are typically indicative of valuable knowledge. The symmetric nature of the three proxies allows the above process to repeat three times, with each proxy— $f_{\theta_1}(\cdot)$ ,  $f_{\theta_2}(\cdot)$ , and  $f_{\theta_3}(\cdot)$ —taking turns as the pseudo-label generator. This learning cycle promotes the sharing of valuable knowledge among the three symmetric proxies, allowing them to collaboratively improve the ensemble performance in handling out-of-distribution designs.
Figure 2: Meta-learning-based sample reweighting.
Despite the efforts made in the first step, small-loss data may still contain inaccurate labels. During the first step, small-loss data  $(p_a$  and  $p_b)$  from the pseudo-labeled dataset produced by  $f_{\theta_1}(\cdot)$  are identified based on the predictions of proxy  $f_{\theta_3}(\cdot)$  and fed to proxy  $f_{\theta_2}(\cdot)$ . However, as shown in Figure 2 (a), the less accurate point  $p_b$  deviates noticeably from the ground-truth, causing the fine-tuned proxy  $f_{\theta_2}(\cdot)$  to diverge from the ground-truth function. To address this, we introduce the second step of ICT, meta-learning-based sample reweighting, which aims to assign higher weights to more accurate points like  $p_a$  and lower weights to less accurate ones like  $p_b$ . To accomplish this, we assign an importance weight for every sample yielded by the first step ( $\omega_a$  for  $p_a$  and  $\omega_b$  for  $p_b$ ) and propose a meta-learning framework to update these sample weights ( $\omega_a$  and  $\omega_b$ ) automatically by leveraging the supervision signals from the offline dataset  $p_{1,2,3}$ . Specifically, the proxy fine-tuned on the weighted small-loss data ( $p_a$  and  $p_b$ ) is expected to perform well on the offline dataset, provided the weights are accurate, i.e., large  $\omega_a$  and small  $\omega_b$ . We can optimize the sample weights by minimizing the loss on the offline dataset as a function of the sample weights. As illustrated in Figure 2 (b), the weight of  $p_a$  is optimized to be high, while the weight of  $p_b$  is optimized to be low. Consequently, the proxy  $f_{\theta_2}^{(b)}(\cdot)$  fine-tuned on the weighted samples in Figure 2 (b) is brought closer to the ground-truth objective function  $f(\cdot)$ , compared to the case where the fine-tuned proxy  $f_{\theta_2}^{(a)}(\cdot)$  is far from  $f(\cdot)$  in Figure 2 (a). Through extensive experiments across various tasks [1], ICT proves effective at mitigating out-of-distribution issues, delivering state-of-the-art results.
In summary, our paper presents three main contributions:
- We introduce Importance-aware Co-Teaching (ICT) for offline MBO. ICT consists of two steps. In the pseudo-label-driven co-teaching step, a proxy is iteratively chosen as the pseudo-labeler, initiating a co-teaching process that facilitates knowledge exchange between the other two proxies.
- The second step, meta-learning-based sample reweighting, is introduced to alleviate potential inaccuracies in pseudo-labels. In this step, pseudo-labeled samples are assigned importance weights, which are then optimized through meta-learning.
- Extensive experiments demonstrate ICT's effectiveness in addressing out-of-distribution issues, yielding state-of-the-art results in multiple MBO tasks. Specifically, ICT secures the best mean rank of 3.1 and median rank of 2, among 15 methods.
Offline model-based optimization (MBO) targets a variety of optimization problems with the goal of maximizing an unknown objective function using an offline dataset. Consider the design space  $\mathcal{X} = \mathbb{R}^d$ , where  $d$  represents the design dimension. Formally, the offline MBO can be expressed as:
$$
\boldsymbol {x} ^ {*} = \underset {\boldsymbol {x} \in \mathcal {X}} {\arg \max } f (\boldsymbol {x}), \tag {1}
$$
where  $f(\cdot)$  denotes the unknown objective function, and  $\pmb{x} \in \mathcal{X}$  denotes a candidate design. In this scenario, an offline dataset  $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N$  is available, where  $x_i$  represents a specific design, such as robot size, and  $y_i$  represents the corresponding score, like robot speed. In addition to robot design, similar problems also include protein and molecule design.
A common strategy for tackling offline MBO involves approximating the unknown objective function  $f(\cdot)$  using a proxy function, typically represented by a deep neural network (DNN)  $f_{\theta}(\cdot)$ , which is trained on the offline dataset:
$$
\boldsymbol {\theta} ^ {*} = \arg \min  _ {\boldsymbol {\theta}} \frac {1}{N} \sum_ {i = 1} ^ {N} \left(f _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {i}\right) - y _ {i}\right) ^ {2}. \tag {2}
$$
With the trained proxy, design optimization is performed using gradient ascent steps:
$$
\boldsymbol {x} _ {t} = \boldsymbol {x} _ {t - 1} + \eta \nabla_ {\boldsymbol {x}} f _ {\boldsymbol {\theta}} (\boldsymbol {x}) \Big | _ {\boldsymbol {x} = \boldsymbol {x} _ {t}}, \quad \text {f o r} t \in [ 1, T ]. \tag {3}
$$
Here,  $T$  denotes the number of steps, and  $\eta$  signifies the learning rate. The optimal design  $\pmb{x}^*$  is acquired as  $\pmb{x}_T$ . This gradient ascent approach is limited by an out-of-distribution issue, as the proxy  $f_{\theta}(\pmb{x})$  may not accurately predict scores for unseen designs, leading to suboptimal solutions.
In this section, we introduce Importance-aware Co-Teaching (ICT), which consists of two steps. We maintain three symmetric proxies and compute the mean ensemble as the final proxy. In Sec 3.1, we describe the first step, pseudo-label-driven co-teaching. This step involves iteratively selecting one proxy as the pseudo-label generator and implementing a co-teaching process to facilitate the exchange of valuable data between the remaining two proxies. Nevertheless, the samples exchanged during co-teaching might still contain inaccurate labels, which necessitates the second step meta-learning-based sample reweighting in Sec 3.2. During this step, each sample from the previous step is assigned an importance weight and updated via meta-learning. Intuitively, the ICT process can be likened to an enhanced paper peer review procedure between three researchers preparing for submission. Each researcher, acting as an author, presents his/her paper to the other two. These two serve as reviewers and co-teach each other important points to better comprehend the paper, ultimately providing their feedback to the author. A detailed depiction of the entire algorithm can be found in Algorithm 1.
3.1 Pseudo-label-driven Co-teaching
Vanilla gradient ascent, as expressed in Eq. (3), is prone to out-of-distribution issues in offline model-based optimization. One potential yet unexplored solution is using a pseudo-labeler to provide pseudo-labels to designs around the optimization point. By fine-tuning the proxy using the valuable portion of the pseudo-labeled dataset, we can enhance the proxy's performance. To achieve this, we maintain three proxies simultaneously, computing their mean ensemble as the final proxy, and iteratively select one proxy to generate pseudo-labeled data. The other two proxies exchange knowledge estimated to have high value, by sharing small-loss data. Due to the symmetric nature of the three proxies, this process can be repeated three times for sharing valuable knowledge further.
Pseudo-label. We initially train three proxies  $f_{\theta_1}(\cdot), f_{\theta_2}(\cdot)$ , and  $f_{\theta_3}(\cdot)$  on the whole offline dataset using Eq. (2) with different initializations, and conduct gradient ascent with their mean ensemble,
$$
\boldsymbol {x} _ {t} = \boldsymbol {x} _ {t - 1} + \eta \nabla_ {\boldsymbol {x}} \frac {1}{3} \left(f _ {1} \left(\boldsymbol {x} _ {t - 1}\right) + f _ {2} \left(\boldsymbol {x} _ {t - 1}\right) + f _ {3} \left(\boldsymbol {x} _ {t - 1}\right)\right), \tag {4}
$$
where  $\eta$  is the gradient ascent learning rate. Given the current optimization point  $\pmb{x}_t$ , we sample  $M$  points  $\pmb{x}_{t,1},\pmb{x}_{t,2},\dots,\pmb{x}_{t,M}$  around  $\pmb{x}_t$  as  $\pmb{x}_{t,m} = \pmb{x}_t + \gamma \epsilon$ , where  $\gamma$  is the noise coefficient and  $\epsilon$  is drawn from the standard Gaussian distribution. An alternative way is to directly sample the  $M$  points around the offline dataset, rather than the current optimization point. We detail this option in Appendix A.1. We iteratively choose one proxy, for example  $f_{\theta_1}(\cdot)$ , to label these points, creating a pseudo-labeled dataset  $\mathcal{D}_1 = \{(x_{t,j},f_{\theta_1}(x_{t,j}))\}_{j = 1}^M$ . Lines 5 to 6 of Algorithm 1 detail the implementation of this segment.
Co-teaching. For each sample in the pseudo-labeled dataset  $\mathcal{D}_1$ , we compute the sample loss for  $f_{\theta_2}(\cdot)$  and  $f_{\theta_3}(\cdot)$ . Specifically, the losses are calculated as  $\mathcal{L}_{2,i} = (f_{\theta_2}(\boldsymbol{x}_{t,i}) - f_{\theta_1}(\boldsymbol{x}_{t,i}))^2$  and  $\mathcal{L}_{3,i} = (f_{\theta_3}(\boldsymbol{x}_{t,i}) - f_{\theta_1}(\boldsymbol{x}_{t,i}))^2$ , respectively. Small-loss samples typically contain valuable knowledge, making them ideal for enhancing proxy robustness [13]. Proxies  $f_{\theta_2}(\cdot)$  and  $f_{\theta_3}(\cdot)$  then exchange the top  $K$  small-loss samples as valuable data to teach each other where  $K$  is a hyperparameter. The co-teaching process enables the exchange of valuable knowledge between proxies  $f_{\theta_2}(\cdot)$  and  $f_{\theta_3}(\cdot)$ . This part is implemented as described in Lines 7 to 8 of Algorithm 1. The symmetric design of the three proxies,  $f_{\theta_1}(\cdot), f_{\theta_2}(\cdot)$ , and  $f_{\theta_3}(\cdot)$ , enables the entire process to be iterated three times with one proxy chosen as the pseudo-labeler every time.
3.2 Meta-learning-based Sample Reweighting
While the previous step effectively selects samples for fine-tuning, these samples may still contain inaccuracies. To mitigate this, we introduce a meta-learning-based sample reweighting step. In this step, each sample obtained from the prior step is assigned an importance weight, which is then updated using a meta-learning framework. Without loss of generality, we use  $f_{\theta}(\cdot)$  to represent any of  $f_{\theta_1}(\cdot), f_{\theta_2}(\cdot)$  and  $f_{\theta_3}(\cdot)$  as this step applies identically to all three proxies. The top  $K$  small-loss samples selected from the previous step for fine-tuning  $f_{\theta}(\cdot)$  are denoted as  $\mathcal{D}_s = \{(x_i^s,\bar{y}_i^s)\}_{i = 1}^K$ .
Sample Reweighting. We assign an importance weight  $\omega_{i}$  to the  $i^{th}$  selected sample and initialize these importance weights to ones. We expect smaller importance weights for less accurate samples
and larger importance weights for more accurate samples to improve proxy fine-tuning. With these weights, we can optimize the proxy parameters as follows:
$$
\boldsymbol {\theta} ^ {*} (\omega) = \arg \min  _ {\boldsymbol {\theta}} \frac {1}{K} \sum_ {i = 1} ^ {K} \omega_ {i} \left(f _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {i} ^ {s}\right) - \bar {y} _ {i} ^ {s}\right) ^ {2}. \tag {5}
$$
Since we only want to perform fine-tuning based on  $\mathcal{D}_s$ , we can adopt one step of gradient descent:
$$
\boldsymbol {\theta} ^ {*} (\omega) = \boldsymbol {\theta} - \frac {\alpha}{K} \sum_ {i = 1} ^ {K} \omega_ {i} \frac {\partial \left(f _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {i} ^ {s}\right) - \bar {y} _ {i} ^ {s}\right) ^ {2}}{\partial \boldsymbol {\theta} ^ {\top}}, \tag {6}
$$
where  $\alpha$  is the learning rate for fine-tuning. This part is presented in Line 10 in Algorithm 1.
Meta-learning. The challenge now is finding a group of proper weights  $\omega$ . We achieve this by leveraging the supervision signals from the offline dataset, which are generally accurate. If the sample weights are accurate, the proxy fine-tuned on the weighted samples is expected to perform well on the offline dataset. This is because the weighted samples aim to reflect the underlying ground-truth function that the offline dataset already captures, and both sets of data share common patterns. We can optimize the sample weights by minimizing the loss of the offline dataset in a meta-learning framework. The loss on the offline dataset can be written as:
$$
\mathcal {L} \left(\boldsymbol {\theta} ^ {*} (\boldsymbol {\omega})\right) = \arg \min  _ {\boldsymbol {\omega}} \frac {1}{N} \sum_ {i = 1} ^ {N} \left(f _ {\boldsymbol {\theta} ^ {*} (\boldsymbol {\omega})} \left(\boldsymbol {x} _ {i}\right) - y _ {i}\right) ^ {2}. \tag {7}
$$
The sample weight  $\omega_{i}$  for the  $i^{th}$  sample can be updated by gradient descent:
$$
\begin{array}{l} \boldsymbol {\omega} _ {i} ^ {\prime} = \boldsymbol {\omega} _ {i} - \beta \frac {\partial \mathcal {L} \left(\boldsymbol {\theta} ^ {*} (\boldsymbol {\omega})\right)}{\partial \boldsymbol {\theta}} \frac {\partial \boldsymbol {\theta} ^ {*} (\boldsymbol {\omega})}{\partial \boldsymbol {\omega} _ {i}} \tag {8} \\ = \omega_ {i} + \frac {\alpha \beta}{K} \frac {\partial \mathcal {L} \left(\boldsymbol {\theta} ^ {*} (\boldsymbol {\omega})\right)}{\partial \boldsymbol {\theta}} \frac {\partial \left(f _ {\boldsymbol {\theta}} \left(\boldsymbol {x} _ {i} ^ {s}\right) - \bar {y} _ {i} ^ {s}\right) ^ {2}}{\partial \boldsymbol {\theta} ^ {\top}}, \\ \end{array}
$$
where  $\beta$  is the learning rate for the meta-learning framework. From Eq. (8), it is worth mentioning that  $\frac{\partial\mathcal{L}(\boldsymbol{\theta}^{*}(\omega))}{\partial\boldsymbol{\theta}}\frac{\partial(f_{\boldsymbol{\theta}}(\boldsymbol{x}_{i}^{s}) - \bar{y}_{i}^{s})^{2}}{\partial\boldsymbol{\theta}^{\top}}$  represents the similarity between the gradient of the offline dataset and the gradient of the  $i^{th}$  sample. This implies that a sample with a gradient similar to the offline dataset will receive a higher weight and vice versa, revealing the inner mechanism of this framework. By applying the updated sample weights to Eq. (6) for fine-tuning, we improve the proxy's performance. This process is iteratively applied to each proxy, yielding a stronger ensemble. Lines 11 to 13 of Algorithm 1 showcase the execution of this part.
4.1 Dataset and Evaluation
Dataset and Tasks. In this study, we conduct experiments on four continuous tasks and three discrete tasks. The continuous tasks include: (a) Superconductor (SuperC)[5], where the objective is to develop a superconductor with 86 continuous components to maximize critical temperature, using 17,010 designs; (b) Ant Morphology (Ant)[1, 14], where the aim is to design a quadrupedal ant with 60 continuous components to improve crawling speed, based on 10,004 designs; (c) D'Kitty Morphology (D'Kitty)[1, 15], where the focus is on shaping a quadrupedal D'Kitty with 56 continuous components to enhance crawling speed, using 10,004 designs; (d) Hopper Controller (Hopper)[1], where the aim is to identify a neural network policy with 5,126 weights to optimize return, using 3,200 designs. Additionally, our discrete tasks include: (e) TF Bind 8 (TF8)[6], where the goal is to discover an 8-unit DNA sequence that maximizes binding activity score, utilizing 32,898 designs; (f) TF Bind 10 (TF10)[6], where the aim is to find a 10-unit DNA sequence that optimizes binding activity score, using 50,000 designs; (g) NAS [16], where the objective is to find the optimal neural network architecture to enhance test accuracy on the CIFAR-10 [17] dataset, using 1,771 designs.
Evaluation and Metrics. In accordance with the evaluation protocol used in [1, 11], we identify the top 128 designs from the offline dataset for each approach and report the  $100^{th}$  percentile normalized
Algorithm 1 Importance-aware Co-teaching
Input: Proxies  $f_{\theta_1}, f_{\theta_2}, f_{\theta_3}$  with parameters  $\theta_1, \theta_2, \theta_3$ ; Offline dataset  $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N$ . Output: High-scoring design  $\boldsymbol{x}^*$ .  
1 /* Start the main steps of ICT */  
2  $\boldsymbol{x}_0$  — the design with highest score in  $\mathcal{D}$ .  
3 for t = 1, 2, ..., T do  
4 /* Pseudo-label-driven Co-teaching */  
5 Sample M points  $\boldsymbol{x}_{t,1}, \boldsymbol{x}_{t,2}, \ldots, \boldsymbol{x}_{t,M}$  around  $\boldsymbol{x}_t$ .  
6 Choose a proxy, such as  $f_{\theta_1}(\cdot)$ , to pseudo-label:  $\mathcal{D}_1 \leftarrow \{(x_{t,j}, f_{\theta_1}(x_{t,j}))\}_{j=1}^M$ .  
7 Identify the top K small-loss samples in  $\mathcal{D}_1$  using  $f_{\theta_3}(\cdot)$  and feed them to  $f_{\theta_2}(\cdot)$ .  
8 Identify the top K small-loss samples in  $\mathcal{D}_1$  using  $f_{\theta_2}(\cdot)$  and feed them to  $f_{\theta_3}(\cdot)$ .  
9 /* Meta-learning-based Sample Reweighting */  
10 Initialize the sample weights  $\omega^{(2)}, \omega^{(3)}$  as ones and perform fine-tuning with Eq. 6).  
11 Update  $\omega^{(2)}$  for  $f_{\theta_2}(\cdot)$  and  $\omega^{(3)}$  for  $f_{\theta_3}(\cdot)$  with Eq. (8).  
12 Update  $\theta_2$  using updated weights  $\omega^{(2)}$  and  $\theta_3$  for  $\omega^{(3)}$  with Eq. (6).  
13 Repeat Line 6 - 12 with  $f_{\theta_2}(\cdot), f_{\theta_3}(\cdot)$  iteratively as pseudo-labeler in Line 6.  
14 /* Finished ICT, start to optimize designs */  
15 for t = 1, 2, ..., T do  
16 Optimize  $\boldsymbol{x}_t$  with Eq. (4) based on the mean ensemble of fine-tuned  $f_{\theta_1}, f_{\theta_2}, f_{\theta_3}$ .  
17 return  $\boldsymbol{x}^* \leftarrow \boldsymbol{x}_T$
ground-truth score. This score is computed as  $y_{n} = \frac{y - y_{min}}{y_{max} - y_{min}}$ , where  $y_{min}$  and  $y_{max}$  represent the minimum and maximum scores within the entire unobserved dataset, respectively. The  $50^{th}$  percentile (median) normalized ground-truth scores are included in Appendix A.2. For a better comparison, we report the best design in the offline dataset, denoted as  $\mathcal{D}(\text{best})$ . We also provide mean and median rankings across all seven tasks for a broad performance assessment.
4.2 Comparison Methods