Five Properties of Specific Curiosity You Didn't Know Curious Machines Should Have

Curiosity for machine agents has been a focus of lively research activity. The study of human and animal curiosity, particularly specific curiosity, has unearthed several properties that would offer important benefits for machine learners, but that have not yet been well-explored in machine intelligence. In this work, we conduct a comprehensive, multidisciplinary survey of the field of animal and machine curiosity. As a principal contribution of this work, we use this survey as a foundation to introduce and define what we consider to be five of the most important properties of specific curiosity: 1) directedness towards inostensible referents, 2) cessation when satisfied, 3) voluntary exposure, 4) transience, and 5) coherent long-term learning. As a second main contribution of this work, we show how these properties may be implemented together in a proof-of-concept reinforcement learning agent: we demonstrate how the properties manifest in the behaviour of this agent in a simple non-episodic grid-world environment that includes curiosity-inducing locations and induced targets of curiosity. As we would hope, our example of a computational specific curiosity agent exhibits short-term directed behaviour while updating long-term preferences to adaptively seek out curiosity-inducing situations. This work, therefore, presents a landmark synthesis and translation of specific curiosity to the domain of machine learning and reinforcement learning and provides a novel view into how specific curiosity operates and in the future might be integrated into the behaviour of goal-seeking, decision-making computational agents in complex environments. ar X iv :2 21 2. 00 18 7v 1 [ cs . A I] 1 D ec 2 02 2 Ady, Shariff, Günther, Pilarski 1. Books, Bookstores, and Machine Curiosity Imagine you have a favourite corner bookstore near your home. You walk into the shop, browse the shelves for something new, take it home and read it end-to-end in less than a week—perhaps at the expense of sleep. The reading feels good; the unfolding plot makes you almost unable to put down the book. You read the last page and want more. There isn’t any more book left, so you walk directly back to the bookstore for a chance at another great read. You don’t buy and read the same book (that would be silly and you know how it ends); instead, you know that the bookstore can give you a new engaging reading experience. The more you read, the more you like reading—and that corner bookstore too! This example is rooted in the properties of human curiosity. In this paper, we focus on improving the specificity of how we think about curiosity with the goal of facilitating the implementation of key properties of human curiosity in machines. Humans have thought about their own curiosity for thousands of years, dating back at least to Aristotle in 350 BCE (Loewenstein, 1994, p. 76, in reference to Metaphysics, Bk. 1, Ch. 2). The study of human curiosity remains an active area of research with many diverse interpretations; recent psychological, neuroscientific and philosophical accounts by Kidd and Hayden (2015) and Zurn (2015, pp. 1–28) review some of this diversity of thought. Over the last three decades, curiosity has started to also catch the focused attention of researchers seeking to create increasingly intelligent non-human machines. Select ideas from the study of human curiosity inspired fantastic breakthroughs in machine intelligence, from a robot dog shifting its own learning focus across progressively more difficult situations (Oudeyer et al., 2007) to a simulated agent achieving higher-than-ever-before scores in Montezuma’s Revenge, one of the so-called “hard exploration” games in the Atari suite (Bellemare et al., 2016, pp. 4, 7; Dvorsky, 2016; Burda et al., 2019, pp. 1,4; Cobbe et al., 2018). Work on machine curiosity is expected to continue to play an influential role in machine intelligence research. Advancing curiosity in the domain of machine intelligence is also expected to have substantial reciprocal benefits for research domains focused on human and animal curiosity, like those in psychology, education, philosophy, neuroscience, and behavioural economics.1 To readers focused on curiosity in those domains: this paper is in large part addressed to you. One goal of this paper is to provide a new perspective on curiosity applicable to any learner, whether human, animal, or machine. Implementing any concept as an algorithm requires a different way of thinking. The abstractions that have been used thus far to improve our understanding of biological curiosity are different from those needed to build a curious machine. This approach and this work provide a unique perspective that may help researchers from multiple disciplines understand curiosity more deeply. This paper is meant to contribute as much to the field of curiosity studies (see Zurn and Shankar, 2020, p. xii–xiii) as to that of machine intelligence. The synthesis completed in this work has led us to define and explore five key properties needed to capture the full value of human curiosity for machine curiosity. While existing 1. Advances in machine intelligence have long supported the development of new theories of biological intelligence (Newell, 1970, pp. 363–370; Sutton and Barto, 2018, Chs. 14–15). We propose that the understanding of curiosity, as a facet of intelligence, can be similarly bolstered through the development of models of curiosity for machine intelligence. Given curiosity’s political role “equip[ping] us to pursue a more intellectually vibrant and equitable world” (p. xi–xii), scholars like Zurn and Shankar (2020) have emphasized the urgency of transdisciplinary conversation on curiosity.