Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking

1 Introduction . Benchmarks have been critical to driving progress in AI : they provide a standard by which models are measured , they support direct comparisons of different proposals , and they provide clear-cut goals for the research community . This has led to an outpouring of new benchmarks designed not only to evaluate models on new tasks , but also to address weaknesses in existing models [ 49 , 55 , 34 ] , and expose artifacts in existing benchmarks [ 42 , 19 , 25 , 33 , 22 , 37 ] . These efforts are helping to provide us with a more realistic picture of how much progress the field has made . To date , the metrics by which we assess system performance have received much less systematic attention . Even as the benchmarks have changed , the community has continued to rely heavily on accuracy as the sole primary metric . This gives rise to a “ leaderboard culture ” in which accuracy is the only thing that matters , even in contexts in which other pressures – e.g. , compactness , fairness , efficiency – are clearly important [ 60 , 6 , 16 , 63 , 68 , 38 , 2 , 51 , 17 ] . Recently , Ethayarajh and Jurafsky [ 17 ] provided a microeconomic framing of the problem : leaderboard viewers are consumers of models , and each viewer has their own set of preferences : some care only about accuracy , others value compactness in addition to accuracy , and so on . Static leaderboards that only rank by model performance thus have a scoring function that is misaligned with the preferences of most users . Moreover , merely focusing on a single metric , such as accuracy , limits the scope of possible issues ⇤Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . even to consider . In other words , focusing on one standard metric , like accuracy , lets researchers off the hook too much on other pressing issues . Reproducibility , accessibility , and backwards compatibility are important issues as well . There is a reliance on self-reported results with no trust guarantees [ 75 ] , and many state-of-the-art claims are famously difficult to reproduce [ 52 ] . In some cases , the state-of-the-art model can not be accessed without sufficient computational resources and/or technical expertise , and older models are often incomparable to newer ones because they have not been evaluated on the same data . To begin to address these issues , we propose Dynaboard . Dynaboard ’ s evaluation-as-a-service backend allows models to be submitted directly to be evaluated in the cloud , circumventing issues of reproducibility , accessibility , and backwards compatibility . Older models can be compared to newer ones because any model can be evaluated on demand . Dynaboard supports reasoning about many different metrics – not just standard accuracy-style metrics , but also memory use and throughput ( i.e. , inference speed ) on identical hardware , robustness , fairness , and so forth . Building on [ 17 ] , we borrow from microeconomic theory to aggregate these metrics – along with performance – into a Dynascore that is used to rank models . The frontend of Dynaboard is a dynamic leaderboard that allows users to customize the Dynascore by placing more/less weight on a particular metric or dataset . As the user modifies the weights that govern the Dynascore , models are re-ranked in real time . Users can also directly interact with a model via the platform , receiving real-time model predictions that help them understand a model ’ s capabilities and limitations , and allowing the community to find challenging examples where the current state of the art still falls short . Together , this allows users to develop and directly express the complex and interrelated values they have for their models . Although other platforms – CodaLab , DAWNBench [ 10 ] , and ExplainaBoard [ 39 ] , to name a few – address some subset of the problems we describe above , Dynaboard is the first to address all of them in a single end-to-end system . It requires minimal overhead for model creators wishing to submit their model for evaluation , but offers maximal flexibility for users wishing to make finegrained comparisons between models . As state-of-the-art NLP models push the limits of traditional benchmarks , Dynaboard offers a standardized solution for creating the next generation of benchmarks in a manner that allows for the diverse and comprehensive evaluation of model quality . 2 Objectives . Dynaboard aims to address the following issues in our current model evaluation paradigm : Reproducibility The reliance on self-reported results with no trust guarantees [ 75 ] makes many state-of-the-art claims difficult to reproduce [ 52 ] . Even the choice of random seed can lead to substantially different results [ 15 ] , and improvements on the state-of-the-art are often not statistically significant [ 8 ] . Implementational differences in evaluation metrics can also lead to different scores [ 54 ] . Given that the test data for static benchmarks is often publicly available , reported results may also be the outcome of overfitting hyperparameters [ 4 ] . Accessibility Whichever model happens to hold the “ state-of-the-art ” title should be accessible by as many researchers as possible . Democratizing model evaluation is essential to understanding our current weaknesses , for making progress in the long tails of the data distribution , and for collecting new adversarial datasets where the state-of-the-art fails . Just open sourcing the model weights is insufficient , because even when given trained models , inference-time compute resources may not be readily available in many parts of the world . Backwards Compatibility Dynamic human-in-the-loop data collection is a great alternative to static benchmarks , but as new evaluation sets are introduced in later rounds , old models become incomparable to new ones , simply because they haven ’ t been evaluated on the same data . It should be possible to evaluate a model on demand , rather than the model ’ s predictions at a single point in time . Forwards Compatibility Automated metrics such as BLEU and ROUGE have many known flaws , and creating better automated metrics ( e.g. , BLEURT [ 64 ] , BERTScore [ 77 ] ) is an active area of research . However , the current paradigm does not allow for old models to be evaluated on new automated metrics that come later , since it relies on reporting from the model creator . If our field comes up with new and better metrics , we should be able to immediately gauge overall model quality . Prediction Costs Most leaderboards treat the cost of making predictions as zero , which does not hold in practice ; memory use , latency , throughput , and lack of robustness – among many other factors – have real-world implications in deployment [ 60 , 17 ] . A highly accurate model may be useless to an embedded systems engineer if it is untenably large , for example . For users to be able to make informed decisions about the risks and rewards of using a particular model , they need to be given as much information as possible . Utility Estimation There is no single correct way to rank models : every leaderboard viewer has a different set of preferences , as manifested in their utility function [ 17 ] . Some users only care about model performance , but others care about memory use , throughput , fairness , and more . A leaderboard should not be overly prescriptive when ranking models – it should provide a default ranking , but ultimately give users the freedom to customize the scoring/ranking function to better approximate their utility function . The evaluation-as-a-service backend of Dynaboard addresses the first four of these . By having models submitted directly to the platform , they can be accessed and evaluated on demand on any dataset ( see Section 4 ) . The frontend of Dynaboard is a dynamic leaderboard that addresses the last two issues : prediction costs and utility estimation . Although the leaderboard has a default ranking , the scoring function is constructed in a principled way , borrowing from economic theory to estimate the rates at which users are willing to trade-off prediction costs for performance . More importantly , users are given the freedom to manipulate the scoring function to better approximate their preferences . 3 Related Work . Evaluation platforms Although other platforms have addressed some subset of the issues in Section 2 , they have not addressed all of them in a single end-to-end solution . DAWNBench [ 10 ] reports some prediction costs , such as the time needed to reach a particular accuracy . However , it still relies on self-reported results , meaning that reproducibility , accessibility , and forwards/backwards compatibility are still issues . CodaLab2 and EvalAI3 make it significantly easier to access models and reproduce stated claims , but automatically evaluating any model on any new dataset or metric is not straightforward , making backwards/forwards compatibility hard to scale . It would also be difficult to collect prediction costs on standard hardware . Explainaboard [ 39 ] and Robustness Gym [ 23 ] allow for a more nuanced comparison of models , but through the lens of which slices of data a model performs well on and what types of errors it makes . Models are not directly uploaded , however , so prediction costs are not collected and forwards compatibility can not be automated . We see these approaches as complementary to Dynaboard , helping provide a richer view of model performance . Multi-metric evaluation There have been some attempts to rank models by metrics other than accuracy . Although not explicitly framed in terms of utility , Mieno et al . studied the trade-off users were willing to make between accuracy and latency in speech translation [ 44 ] . Some challenge tasks in recent years have required the reporting of prediction costs or have instituted limits on the maximum costs that can be incurred , as in the EfficientQA challenge [ 27 , 45 ] . However , such practices are still few and far between . Our hope is that over time Dynaboard can help lower the overhead required to create multi-metric benchmarks and help them proliferate and diversify , such that every sub-community can have its own canonical benchmarks , like GEM [ 21 ] for natural language generation , BIG-Bench4 for language model probing , or GLUE [ 73 ] and SuperGLUE [ 72 ] for NLU . 4 Backend : Evaluation-as-a-Service . The evaluation-as-a-service pipeline uses a standardized application programming interface ( API ) that is similar to other model evaluation platforms like EvalAI and Codalab that allow creators to upload models . Researchers submit their models for evaluation in our model evaluation cloud , in which all models are scored in exactly the same way , on the exact same data , within the same ( virtual ) hardware constraints . Testing models on different datasets , from standard well-known test sets to 2https : //codalab.org 3https : //eval.ai 4https : //github.com/google/BIG-bench stress tests [ 47 , 11 ] and check lists [ 59 ] can be cumbersome – we offer evaluation-as-a-service ( EaaS ) for NLP models . We ask model developers to specify model cards [ 46 ] to report on and properly document their contributions . In this paradigm , it is no longer possible for people to cherrypick results : the same model instance is evaluated on all evaluation sets . We take a multi-metric approach , going beyond just accuracy , where models are evaluated on multiple axes of evaluation . The top models can be employed in-the-loop for dynamic adversarial evaluation [ 34 ] , and all uploaded models can be made accessible online so that anyone can “ talk to ” these models ( in a corresponding frontend UI ) to see how they perform . Models are evaluated on combinations of adversarial and non-adversarial datasets . All statistics are made available to the user , who can rank models by specifying which datasets and which metrics should factor into the ranking and in what proportion ( see Section 5 for details ) . Importantly , we do not claim to have built or discovered the one right approach for model evaluation . The goal of Dynaboard – and the Dynabench data collection platform it is built on – is making dynamic as many of the components employed by the current status quo as possible : i ) rather than having a single metric , we can have many metrics , with the ability to add more as new automated metrics are conceived ; ii ) rather than having a fixed set of evaluation datasets , we can have many and add more over time ; iii ) rather than evaluating models only on static datasets , they can be evaluated dynamically in the loop . For this approach to succeed , the ability to re-evaluate older models is essential : when new datasets are introduced , new metrics come out , and as new community preferences emerge , we need to be able to see where we stand with our current state-of-the-art .

Summary: The paper proposes an evaluation platform "DynaBoard" to automatic evaluate the models for different NLP tasks according to 5 metrics: performance, throughput, memory, fairness, and robustness. The scores from these metrics can be aggregated according to the custom preferences of each user. The models can then be ranked according to the aggregated score.