A Continuous Information Gain Measure to Find the Most Discriminatory Problems for AI Benchmarking
Channel: Christoph Salge
Date: 2020-08-25

Behind this exciting title hides a talk about one of our latest scientific papers - where we discuss how to design a measure that can pick the most discriminatory games from a big set of games in an AI benchmark. I apologize for the mediocre video quality, but what can I say, I am a scientist, not a vlogger. This might be mostly interesting if you read our paper, and want some more explanation, maybe with some illustrations. So, if you got here by accident, I attached a link to the paper, you can go, read it, and then come back. 

Get paper: https://arxiv.org/abs/1809.02904

Abstract: Matthew Stephenson, Damien Anderson, Ahmed Khalifa, John Levine, Jochen Renz, Julian Togelius, Christoph Salge.

This paper introduces an information-theoretic method for selecting a subset of problems which gives the most information about a group of problem-solving algorithms. This method was tested on the games in the General Video Game AI (GVGAI) framework, allowing us to identify a smaller set of games that still gives a large amount of information about the abilities of different game-playing agents. This approach can be used to make agent testing more efficient. We can achieve almost as good discriminatory accuracy when testing on only a handful of games as when testing on more than a hundred games, something which is often computationally infeasible. Furthermore, this method can be extended to study the dimensions of the effective variance in game design between these games, allowing us to identify which games differentiate between agents in the most complementary ways.