text: 200,162,1400877275357270016,54910963,Clara Vania,"Many NLU datasets have been created to evaluate various aspects of language, but which datasets are still effective to measure future progress? Check out our new paper, to appear at #acl2021nlp #NLProc <LINK> (1/8) <LINK> Taking inspiration from psychometric studies which often use Item Response Theory (IRT) to evaluate test items in educational assessment, we apply it to evaluate test examples from 29 English datasets. (2/8) We use predictions from 18 Transformer-based models with varying degrees of abilities, and estimate how difficult and discriminative a test example is relative to other examples. (3/8) We introduce a new metric called Locally Estimated Headroom (LEH) to estimate how much a dataset is still useful to measure near-future progress. We find that Quoref, HellaSwag, and MC-TACO are still effective, while SNLI, MNLI, and BoolQ seem to be saturated. (4/8) <LINK> We also find that span-based QA is the most effective task format to discriminate between strong and weak models. (5/8) However, datasets that contain many discriminative examples do not always have examples that are the most difficult. SQuAD2.0, QuAIL, and ANLI appear to have many examples with the highest difficulty levels. (6/8) <LINK> Please see the paper for more details. We hope this work can give insights to future dataset creation and model development in NLP, and we argue that this evaluation should be done periodically over time. (7/8) Joint work with amazing co-authors: @phu_pmh, @WillHuang93, Dhara Mungra, @yzpang97, @zhansheng, @liu_haokun, @kchonyc, and @sleepinyourhat. (8/8)",https://arxiv.org/abs/2106.00840,"Recent years have seen numerous NLP datasets introduced to evaluate the performance of fine-tuned models on natural language understanding tasks. Recent results from large pretrained models, though, show that many of these datasets are largely saturated and unlikely to be able to detect further progress. What kind of datasets are still effective at discriminating among strong models, and what kind of datasets should we expect to be able to detect future improvements? To measure this uniformly across datasets, we draw on Item Response Theory and evaluate 29 datasets using predictions from 18 pretrained Transformer models on individual test examples. We find that Quoref, HellaSwag, and MC-TACO are best suited for distinguishing among state-of-the-art models, while SNLI, MNLI, and CommitmentBank seem to be saturated for current strong models. We also observe span selection task format, which is used for QA datasets like QAMR or SQuAD2.0, is effective in differentiating between strong and weak models. ",Comparing Test Sets with Item Response Theory,8,"['Many NLU datasets have been created to evaluate various aspects of language, but which datasets are still effective to measure future progress? Check out our new paper, to appear at #acl2021nlp #NLProc  <LINK> (1/8) <LINK>', 'Taking inspiration from psychometric studies which often use Item Response Theory (IRT) to evaluate test items in educational assessment, we apply it to evaluate test examples from 29 English datasets. (2/8)', 'We use predictions from 18 Transformer-based models with varying degrees of abilities, and estimate how difficult and discriminative a test example is relative to other examples. (3/8)', 'We introduce a new metric called Locally Estimated Headroom (LEH) to estimate how much a dataset is still useful to measure near-future progress. We find that Quoref, HellaSwag, and MC-TACO are still effective, while SNLI, MNLI, and BoolQ seem to be saturated. (4/8) https://t.co/N0fw6xF6d7', 'We also find that span-based QA is the most effective task format to discriminate between strong and weak models. (5/8)', 'However, datasets that contain many discriminative examples do not always have examples that are the most difficult. SQuAD2.0, QuAIL, and ANLI appear to have many examples with the highest difficulty levels. (6/8) https://t.co/l1E9Qgfktz', 'Please see the paper for more details. We hope this work can give insights to future dataset creation and model development in NLP, and we argue that this evaluation should be done periodically over time. (7/8)', 'Joint work with amazing co-authors: @phu_pmh, @WillHuang93, Dhara Mungra, @yzpang97, @zhansheng, @liu_haokun, @kchonyc, and @sleepinyourhat. (8/8)']",21,06,1587