From Word Models to World Models: Translating from Natural Language to
  the Probabilistic Language of Thought
Authors: name: Lionel Wong; name: Gabriel Grand; name: Alexander K. Lew; name: Noah D. Goodman; name: Vikash K. Mansinghka; name: Jacob Andreas; name: Joshua B. Tenenbaum
Published: 2023-06-22 05:14:00

Abstract: How does language inform our downstream thinking? In particular, how do
humans make meaning from language -- and how can we leverage a theory of
linguistic meaning to build machines that think in more human-like ways? In
this paper, we propose rational meaning construction, a computational
framework for language-informed thinking that combines neural models of
language with probabilistic models for rational inference. We frame linguistic
meaning as a context-sensitive mapping from natural language into a
probabilistic language of thought (PLoT) -- a general-purpose symbolic
substrate for probabilistic, generative world modeling. Our architecture
integrates two powerful computational tools that have not previously come
together: we model thinking with probabilistic programs, an expressive
representation for flexible commonsense reasoning; and we model meaning
construction with large language models (LLMs), which support
broad-coverage translation from natural language utterances to code expressions
in a probabilistic programming language. We illustrate our framework in action
through examples covering four core domains from cognitive science:
probabilistic reasoning, logical and relational reasoning, visual and physical
reasoning, and social reasoning about agents and their plans. In each, we show
that LLMs can generate context-sensitive translations that capture
pragmatically-appropriate linguistic meanings, while Bayesian inference with
the generated programs supports coherent and robust commonsense reasoning. We
extend our framework to integrate cognitively-motivated symbolic modules to
provide a unified commonsense thinking interface from language. Finally, we
explore how language can drive the construction of world models themselves.