PANEL 6: METRICS AND MEASUREMENT METHODS: 

WHAT AND HOW TO TEST

José Hernández-Orallo (jorallo@upv.es)

Valencian Research Institute for Artificial Intelligence (vrAIn) (vrain.upv.es)

Universitat Politècnica de València, València (www.upv.es)

Leverhulme Centre for the Future of Intelligence, Cambridge (lcfi.ac.uk)

NIST Workshop on AI Measurement and Evaluation, June 15-17, 2021
N I S T W O R K S H O P O N A I M E A S U R E M E N T A N D E V A L U A T I O N :

W H A T A N D H O W T O T E S T 2

Prose Award 2018
AI EVALUATION AS AGGREGATED PERFORMANCE

▪ GOAL: Estimate the expected result R෨ of system π and a new task μ.

Given:

▪ Distribution p in problem class M (e.g., configurations of a navigation task)

▪ Metric of performance R (e.g., navigation success)

Calculate aggregated performance and extrapolate for μ!

▪ This is useful if μ~p and the operating condition in R does not change.

N I S T W O R K S H O P O N A I M E A S U R E M E N T A N D E V A L U A T I O N :

W H A T A N D H O W T O T E S T 3

But this is almost never the case!
FROM TASK-ORIENTED TO CAPABILITY-ORIENTED EVALUATION

▪ Infers a capability profile for system π. We also use a problem profile for μ.

▪ Given both, estimate

▪ Key ideas:

▪ Instance difficulties become dual to capabilities (à la IRT).

▪ Requires identifying the capabilities and their relation.

▪ Constructs for π and μ are latent factors: measurement is no longer additive.

N I S T W O R K S H O P O N A I M E A S U R E M E N T A N D E V A L U A T I O N :

W H A T A N D H O W T O T E S T 4

Hernández-Orallo, J. (2017). Evaluation in 

artificial intelligence: from task-oriented to 

ability-oriented measurement. Artificial 

Intelligence Review, 48(3), 397-447.

Crosby, M., Beyret, B., 

Shanahan, M., HernándezOrallo, J., Cheke, L., & Halina, 

M. (2020). The animal-ai testbed 

and competition. In NeurIPS

2019 Competition and 

Demonstration Track (pp. 164-

176). PMLR.

Martínez-Plumed, F., Prudêncio, R. 

B., Martínez-Usó, A., & HernándezOrallo, J. (2019). Item response 

theory in AI: Analysing machine 

learning classifiers at the instance 

level. Artificial Intelligence, 271, 18-

42.

F Martinez-Plumed, J Hernandez-Orallo 

(2020) “Dual indicators to analyse AI 

benchmarks: Difficulty, discrimination, 

ability and generality” IEEE Transactions on 

Games. 12(2), 121 - 131

π μ
RECOG-AI : MEASUREMENT LAYOUTS

N I S T W O R K S H O P O N A I M E A S U R E M E N T A N D E V A L U A T I O N :

W H A T A N D H O W T O T E S T 5

▪ Robust Evaluation of Cognitive Capabilities and Generality in AI

▪ 2021-2023 (planning to work with DARPA)

▪ related to the machine common sense program, director: Matt Turek.

▪ Run at the Centre for the Future of Intelligence, Cambridge, UK.

▪ Measurement Layouts:

▪

▪ Generality:

▪ In RL settings for basic 

navigation skills

▪ With language or 

multimodal models
RECOG-AI : SPACES AND FEATURES

▪ Original feature space: 

▪ observable by the system. Usually abstracted 

into latent features.

▪ Surface feature space: 

▪ sometimes observable. A general system should 

be invariant to these.

▪ Cognitive (construct) space: 

▪ usually non-observable. Performance should 

correlate with them: 

▪ agents with a high capabilities profile in this 

space will imply success for problems with 

lower difficulty levels in these capabilities.

N I S T W O R K S H O P O N A I M E A S U R E M E N T A N D E V A L U A T I O N :

W H A T A N D H O W T O T E S T 6

Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., ... & 

Fei-Fei, L. (2015). Imagenet large scale visual recognition challenge. 

International journal of computer vision, 115(3), 211-252.

Textureless

Number

Clutter

128x128 RGB pixels

Symmetry Irrelevant elements
METRICS AND MEASUREMENT

▪ Metrics: 

▪ Capabilities should have a proper scale.

▪ Aggregations are not additive from results.

▪ More detailed results, annotated instances!

▪ No more aggregated results only, please!

▪ No more “superhuman” claims, please!

▪ Measurement: 

▪ Cover the capabilities space, not the original p.

▪ Avoid “challenge-solve-and-replace” dynamics.

▪ Explore instance variation:

▪ Adaptive testing

▪ Adversarial testing

N I S T W O R K S H O P O N A I M E A S U R E M E N T A N D E V A L U A T I O N :

W H A T A N D H O W T O T E S T 7

Hernandez-Orallo, J. “AI Evaluation: On Broken Yardsticks 

and Measurement Scales”, MetaEval@AAAI2020.

“Give me the data 

(distribution) and I will 

ace the test in a year!”

CIFAR10 → CIFAR100, 

SQuAD1.1 → SQuAD2.0, 

GLUE → SUPERGLUE, 

Starcraft → Starcraft II
THANKS!

N I S T W O R K S H O P O N A I M E A S U R E M E N T A N D E V A L U A T I O N :

W H A T A N D H O W T O T E S T 8
OTHER SOURCES AND INITIATIVES: 

9

▪ Other Talks (http://josephorallo.webs.upv.es/)

▪ Diversity Unites Intelligence: Measuring Generality

▪ Measuring A(G)I Right: Some Theoretical and Practical Considerations

▪ Natural and Artificial Intelligence: Measures, Maps and Taxonomies

▪ Book (http://allminds.org):

▪ The Measure of All Minds: Evaluating Natural 

and Artificial Intelligence, Cambridge University Press 2017

▪ The AI Collaboratory: http://aicollaboratory.org/

▪ Part of the European Commission’s AI watch:

▪ https://ec.europa.eu/knowledge4policy/ai-watch_en

▪ ReCOG-AI and the animal AI environment:

▪ Part of the Kinds of Intelligence Programme at the CFI in Cambridge

▪ http://lcfi.ac.uk/projects/kinds-of-intelligence

N I S T W O R K S H O P O N A I M E A S U R E M E N T A N D E V A L U A T I O N :

W H A T A N D H O W T O T E S T