RL Agents That Think Like Children — Exploring Exploration
By: Karthik Bhaskar
Tags: Artificial Intelligence, Machine Learning, Reinforcement Learning, Data Science, Deep Learning

Despite recent progress in artificial intelligence (AI) research, human children are still by far the most skilled learners we know of, learning noble skills like language and high-level thinking from minimal data. An efficient, hypothesis-driven exploration helps children’s learning. In fact, they explore so well that numerous ML researchers have been encouraged to put videos like the one below in their speeches to motivate study into exploration methods. However, because using results from studies in developmental psychology can be challenging, this video is often the degree to which such research correlates with human cognition.

A time-lapse of a baby playing with toys (https://www.youtube.com/watch?v=8vNxjwt2AqY)

Why is directly employing research from developmental psychology to problems in Artificial Intelligence so tricky? Taking motivation from developmental studies can be challenging because the environments that human children and RL agents are typically investigated in can be very diverse. Traditionally, reinforcement learning (RL) investigation takes place in grid-world-like environments or other 2-dimensional games, whereas children act in the rich and 3-dimensional real world. Moreover, similarities between children and AI agents are challenging because the tests are not regulated and often have an objective mismatch; much of the developmental psychology study with children takes place with children involved in the free exploration, whereas most AI research is goal-driven. Lastly, it can be difficult to ‘close the loop’ and build agents inspired by children and learn about human perception from AI research results. By examining children and artificial agents in the identical controlled, 3D environment, we can probably mitigate several of these difficulties above and ultimately advance research in AI and cognitive science.

Recently, Deepmind and Berkeley Artificial Intelligence Research (BAIR) collaborated on a research and developed a directly contrasting agent and child exploration platform based on DeepMind Lab.

How do Children Explore?

The foremost information that we know about infant exploration is that infants form hypotheses about how the world operates, and they engage in exploration to inquire those hypotheses. For instance, studies such as Liz Bonawitz et al., 2007 showed that their observation influences kids’ exploratory play. They decide that if it seems like there are multiple approaches that a plaything could work, but it’s not obvious which one is best (namely, the evidence is causally confounded). Children participate in hypothesis-driven exploration and will investigate the plaything for significantly longer than when the dynamics and result are easy (in which case they would instantly move on to a new plaything).

Stahl and Feigneson et al., 2015 revealed us that when children as young as 11-months are shown with gadgets that disrupt physical laws in their environments, they will investigate them more and even participate in hypothesis-testing behaviours that indicate the particular kind of disruption seen. For instance, if they observe a car floating in the space (as in the video on the top), they discover this unexpected; subsequently, children crash the toy on the desk to explore how it works. In other words, these disruptions significantly lead the children’s exploration.

How do AI Agents Explore?

Standard work in computer science and AI concentrated on improving search methods to investigate a goal. For instance, a depth-first search approach will explore a particular path until either the goal or a state is terminated. If a state is terminated, it will backtrack till the subsequent unexplored path is discovered and then continue down that path. Unlike children’s exploration, practices like these don’t have a concept of exploring more given unexpected evidence, collecting information, or examining hypotheses. Current work in RL has witnessed the development of other varieties of exploration algorithms. For example, intrinsic motivation methods give a reward for exploring engaging regions, such as those that have not been visited as much before or those which are unexpected. While these seem more related to children’s exploration, they are typically practiced more to expose agents to a different set of experiences during training rather than to encourage rapid learning and exploration at decision time.

Conclusion

DeepMind and BAIR did a range of experiments and presented their conclusions on arXiv. Take a look at the paper for a detailed explanation of the experimental setup and results. In conclusion, this work only opens several deep questions about how children and AI agents explore. The experiments are shown here begin to discuss how much children and AI agents are ready to explore, whether free versus goal-directed exploration policies differ, and how reward shaping influences exploration.

The authors of the paper believe that to truly build intelligent agents, they must do as children do: actively explore their environments, perform experiments, and gather information to weave together into a rich model of the world.

In this way, we will understand how children and agents explore novel environments and how to close the gap between them.

Reference

Exploring Exploration: Comparing Children with RL Agents in Unified Environments.

Eliza Kosoy, Jasmine Collins, David M. Chan, Sandy Huang, Deepak Pathak, Pulkit Agrawal, John Canny, Alison Gopnik, and Jessica B. Hamrick arXiv preprint arXiv:2005.02880 (2020)