Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking

1 Introduction . Benchmarks have been critical to driving progress in AI : they provide a standard by which models are measured , they support direct comparisons of different proposals , and they provide clear-cut goals for the research community . This has led to an outpouring of new benchmarks designed not only to evaluate models on new tasks , but also to address weaknesses in existing models [ 49 , 55 , 34 ] , and expose artifacts in existing benchmarks [ 42 , 19 , 25 , 33 , 22 , 37 ] . These efforts are helping to provide us with a more realistic picture of how much progress the field has made . To date , the metrics by which we assess system performance have received much less systematic attention . Even as the benchmarks have changed , the community has continued to rely heavily on accuracy as the sole primary metric . This gives rise to a “ leaderboard culture ” in which accuracy is the only thing that matters , even in contexts in which other pressures – e.g. , compactness , fairness , efficiency – are clearly important [ 60 , 6 , 16 , 63 , 68 , 38 , 2 , 51 , 17 ] . Recently , Ethayarajh and Jurafsky [ 17 ] provided a microeconomic framing of the problem : leaderboard viewers are consumers of models , and each viewer has their own set of preferences : some care only about accuracy , others value compactness in addition to accuracy , and so on . Static leaderboards that only rank by model performance thus have a scoring function that is misaligned with the preferences of most users . Moreover , merely focusing on a single metric , such as accuracy , limits the scope of possible issues ⇤Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . even to consider . In other words , focusing on one standard metric , like accuracy , lets researchers off the hook too much on other pressing issues . Reproducibility , accessibility , and backwards compatibility are important issues as well . There is a reliance on self-reported results with no trust guarantees [ 75 ] , and many state-of-the-art claims are famously difficult to reproduce [ 52 ] . In some cases , the state-of-the-art model can not be accessed without sufficient computational resources and/or technical expertise , and older models are often incomparable to newer ones because they have not been evaluated on the same data . To begin to address these issues , we propose Dynaboard . Dynaboard ’ s evaluation-as-a-service backend allows models to be submitted directly to be evaluated in the cloud , circumventing issues of reproducibility , accessibility , and backwards compatibility . Older models can be compared to newer ones because any model can be evaluated on demand . Dynaboard supports reasoning about many different metrics – not just standard accuracy-style metrics , but also memory use and throughput ( i.e. , inference speed ) on identical hardware , robustness , fairness , and so forth . Building on [ 17 ] , we borrow from microeconomic theory to aggregate these metrics – along with performance – into a Dynascore that is used to rank models . The frontend of Dynaboard is a dynamic leaderboard that allows users to customize the Dynascore by placing more/less weight on a particular metric or dataset . As the user modifies the weights that govern the Dynascore , models are re-ranked in real time . Users can also directly interact with a model via the platform , receiving real-time model predictions that help them understand a model ’ s capabilities and limitations , and allowing the community to find challenging examples where the current state of the art still falls short . Together , this allows users to develop and directly express the complex and interrelated values they have for their models . Although other platforms – CodaLab , DAWNBench [ 10 ] , and ExplainaBoard [ 39 ] , to name a few – address some subset of the problems we describe above , Dynaboard is the first to address all of them in a single end-to-end system . It requires minimal overhead for model creators wishing to submit their model for evaluation , but offers maximal flexibility for users wishing to make finegrained comparisons between models . As state-of-the-art NLP models push the limits of traditional benchmarks , Dynaboard offers a standardized solution for creating the next generation of benchmarks in a manner that allows for the diverse and comprehensive evaluation of model quality . 2 Objectives . Dynaboard aims to address the following issues in our current model evaluation paradigm : Reproducibility The reliance on self-reported results with no trust guarantees [ 75 ] makes many state-of-the-art claims difficult to reproduce [ 52 ] . Even the choice of random seed can lead to substantially different results [ 15 ] , and improvements on the state-of-the-art are often not statistically significant [ 8 ] . Implementational differences in evaluation metrics can also lead to different scores [ 54 ] . Given that the test data for static benchmarks is often publicly available , reported results may also be the outcome of overfitting hyperparameters [ 4 ] . Accessibility Whichever model happens to hold the “ state-of-the-art ” title should be accessible by as many researchers as possible . Democratizing model evaluation is essential to understanding our current weaknesses , for making progress in the long tails of the data distribution , and for collecting new adversarial datasets where the state-of-the-art fails . Just open sourcing the model weights is insufficient , because even when given trained models , inference-time compute resources may not be readily available in many parts of the world . Backwards Compatibility Dynamic human-in-the-loop data collection is a great alternative to static benchmarks , but as new evaluation sets are introduced in later rounds , old models become incomparable to new ones , simply because they haven ’ t been evaluated on the same data . It should be possible to evaluate a model on demand , rather than the model ’ s predictions at a single point in time . Forwards Compatibility Automated metrics such as BLEU and ROUGE have many known flaws , and creating better automated metrics ( e.g. , BLEURT [ 64 ] , BERTScore [ 77 ] ) is an active area of research . However , the current paradigm does not allow for old models to be evaluated on new automated metrics that come later , since it relies on reporting from the model creator . If our field comes up with new and better metrics , we should be able to immediately gauge overall model quality . Prediction Costs Most leaderboards treat the cost of making predictions as zero , which does not hold in practice ; memory use , latency , throughput , and lack of robustness – among many other factors – have real-world implications in deployment [ 60 , 17 ] . A highly accurate model may be useless to an embedded systems engineer if it is untenably large , for example . For users to be able to make informed decisions about the risks and rewards of using a particular model , they need to be given as much information as possible . Utility Estimation There is no single correct way to rank models : every leaderboard viewer has a different set of preferences , as manifested in their utility function [ 17 ] . Some users only care about model performance , but others care about memory use , throughput , fairness , and more . A leaderboard should not be overly prescriptive when ranking models – it should provide a default ranking , but ultimately give users the freedom to customize the scoring/ranking function to better approximate their utility function . The evaluation-as-a-service backend of Dynaboard addresses the first four of these . By having models submitted directly to the platform , they can be accessed and evaluated on demand on any dataset ( see Section 4 ) . The frontend of Dynaboard is a dynamic leaderboard that addresses the last two issues : prediction costs and utility estimation . Although the leaderboard has a default ranking , the scoring function is constructed in a principled way , borrowing from economic theory to estimate the rates at which users are willing to trade-off prediction costs for performance . More importantly , users are given the freedom to manipulate the scoring function to better approximate their preferences . 3 Related Work . Evaluation platforms Although other platforms have addressed some subset of the issues in Section 2 , they have not addressed all of them in a single end-to-end solution . DAWNBench [ 10 ] reports some prediction costs , such as the time needed to reach a particular accuracy . However , it still relies on self-reported results , meaning that reproducibility , accessibility , and forwards/backwards compatibility are still issues . CodaLab2 and EvalAI3 make it significantly easier to access models and reproduce stated claims , but automatically evaluating any model on any new dataset or metric is not straightforward , making backwards/forwards compatibility hard to scale . It would also be difficult to collect prediction costs on standard hardware . Explainaboard [ 39 ] and Robustness Gym [ 23 ] allow for a more nuanced comparison of models , but through the lens of which slices of data a model performs well on and what types of errors it makes . Models are not directly uploaded , however , so prediction costs are not collected and forwards compatibility can not be automated . We see these approaches as complementary to Dynaboard , helping provide a richer view of model performance . Multi-metric evaluation There have been some attempts to rank models by metrics other than accuracy . Although not explicitly framed in terms of utility , Mieno et al . studied the trade-off users were willing to make between accuracy and latency in speech translation [ 44 ] . Some challenge tasks in recent years have required the reporting of prediction costs or have instituted limits on the maximum costs that can be incurred , as in the EfficientQA challenge [ 27 , 45 ] . However , such practices are still few and far between . Our hope is that over time Dynaboard can help lower the overhead required to create multi-metric benchmarks and help them proliferate and diversify , such that every sub-community can have its own canonical benchmarks , like GEM [ 21 ] for natural language generation , BIG-Bench4 for language model probing , or GLUE [ 73 ] and SuperGLUE [ 72 ] for NLU . 4 Backend : Evaluation-as-a-Service . The evaluation-as-a-service pipeline uses a standardized application programming interface ( API ) that is similar to other model evaluation platforms like EvalAI and Codalab that allow creators to upload models . Researchers submit their models for evaluation in our model evaluation cloud , in which all models are scored in exactly the same way , on the exact same data , within the same ( virtual ) hardware constraints . Testing models on different datasets , from standard well-known test sets to 2https : //codalab.org 3https : //eval.ai 4https : //github.com/google/BIG-bench stress tests [ 47 , 11 ] and check lists [ 59 ] can be cumbersome – we offer evaluation-as-a-service ( EaaS ) for NLP models . We ask model developers to specify model cards [ 46 ] to report on and properly document their contributions . In this paradigm , it is no longer possible for people to cherrypick results : the same model instance is evaluated on all evaluation sets . We take a multi-metric approach , going beyond just accuracy , where models are evaluated on multiple axes of evaluation . The top models can be employed in-the-loop for dynamic adversarial evaluation [ 34 ] , and all uploaded models can be made accessible online so that anyone can “ talk to ” these models ( in a corresponding frontend UI ) to see how they perform . Models are evaluated on combinations of adversarial and non-adversarial datasets . All statistics are made available to the user , who can rank models by specifying which datasets and which metrics should factor into the ranking and in what proportion ( see Section 5 for details ) . Importantly , we do not claim to have built or discovered the one right approach for model evaluation . The goal of Dynaboard – and the Dynabench data collection platform it is built on – is making dynamic as many of the components employed by the current status quo as possible : i ) rather than having a single metric , we can have many metrics , with the ability to add more as new automated metrics are conceived ; ii ) rather than having a fixed set of evaluation datasets , we can have many and add more over time ; iii ) rather than evaluating models only on static datasets , they can be evaluated dynamically in the loop . For this approach to succeed , the ability to re-evaluate older models is essential : when new datasets are introduced , new metrics come out , and as new community preferences emerge , we need to be able to see where we stand with our current state-of-the-art .

Summary: This paper introduced Dynaboard, an evaluation framework in which models are uploaded to and evaluated in the cloud in a way that is reproducible and standardized. Doing so makes it easier to track other metrics that are often neglected, such as throughput, memory usage, fairness, and robustness. Finally, the paper introduces Dynascore, a measure that aggregates performance across different metrics, the weights of which can be dynamically chosen by the user to reflect their preferences.