SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Joy Hsu*
Jiayuan Mao*
Joshua B. Tenenbaum
Jiajun Wu
Recent works such as VisProg and ViperGPT have smartly composed foundation models for visual reasoning—using large language models (LLMs) to produce programs that can be executed by pre-trained vision-language models. However, they operate in limited domains, such as 2D images, not fully exploiting the generalization of language: abstract concepts like "left" can also be grounded in 3D, temporal, and action data, as in moving to your left. This limited generalization stems from these inference-only methods' inability to learn or adapt pre-trained models to a new domain. We propose the Logic-Enhanced FoundaTion Model (LEFT), a unified framework that learns to ground and reason with concepts across domains with a differentiable, domain-independent, first-order logic-based program executor. LEFT has an LLM interpreter that outputs a program represented in a general, logic-based reasoning language, which is shared across all domains and tasks. LEFT's executor then executes the program with trainable domain-specific grounding modules. We show that LEFT flexibly learns concepts in four domains: 2D images, 3D scenes, human motions, and robotic manipulation. It exhibits strong reasoning ability in a wide variety of tasks, including those that are complex and not seen during training, and can be easily applied to new domains.
Figure 1: LEFT is a unified concept learning and reasoning framework that grounds modular concepts across domains, and flexibly reasons with concepts across tasks with a foundation model.
The power of language lies in its generalization based on abstraction. A single concept, such as left, can be used across domains: in perception, we recognize the left leg from a chair's 3D point cloud; in navigation, we go home by turning left around the cafe at the end of the block; in manipulation, we pick and place a mug on the left of a plate. Left has domain-specific groundings in each of these
*Equal contribution. Email: joycj@stanford.edu
domains, but the concept symbol itself as an abstraction serves as a basis for complex and multi-step reasoning across domains.
Machine systems for visual reasoning also benefit from the disentanglement of concept learning and reasoning. Most recently, models such as VisProg [Gupta and Kembhavi, 2023] and ViperGPT [Surís et al., 2023] have leveraged large language models (LLMs) to produce programs based on language queries. For example, to answer "what's the color of the cat?", the program will first locate the cat, then query its color. Such programs are executed with pre-defined and pre-trained vision-language models (e.g., open-vocabulary segmentation models) for grounding and question-answering. These works perform well on image-based reasoning, though surprisingly, despite the generalization seemingly offered for free by language, less success has been shown in other domains.
What's left to make visual reasoning systems work across domains? We identify two key shortcomings of existing methods. First, they are inference-only, building exclusively upon pre-trained models to interpret concepts such as left during program execution. Therefore, in domains where data is scarce (e.g., 3D, human motion, or robot action), pretrained models, if there exist any, do not perform as well. Thus, we need trainable systems that learn the domain-specific groundings of concepts to execute across domains. Unfortunately, these visual reasoning models also cannot be made trainable due to their second shortcoming: they rely on a non-differentiable program executor to compose these grounding modules. Therefore, though left may have different groundings in 3D scenes and robot actions compared to 2D images, these modules will not be able to adapt through additional training in those domains.
We address these issues by proposing Logic-Enhanced Foundation Models (LEFT), a model that learns to ground and reason with concepts across domains. LEFT also leverages LLMs for language parsing; but unlike prior work, LEFT has trainable concept grounding modules that learn from domain data, made possible via a differentiable, domain-independent, first-order logic-based program executor. Inspired by previous works that combine symbolic programs and deep models via differentiable execution [Mao et al., 2019, Dong et al., 2019], our design no longer requires manually designed domain-specific languages and facilitates generalization across domains. Specifically, LEFT's LLM interpreter takes as input language queries, resolves textual ambiguities, and outputs a non-ambiguous program represented in a general reasoning language of first-order logic, which is shared across domains and tasks. LEFT's executor then executes the logic program with learnable domain-specific grounding modules, which are automatically initialized with concepts generated by the LLM.
LEFT enjoys the advantages of strong performance and data efficiency across domains due to its modular structure. Within a single domain, LEFT generalizes zero-shot to unseen and complex tasks, by leveraging general reasoning capabilities from the LLM interpreter, and effectively recomposing learned, grounded concepts with the generic first-order logic executor. LEFT can be viewed as a generalized framework of VisProg and ViperGPT; in domains where pre-trained models are available and training is not required (e.g., 2D images), LEFT can similarly be used inference-only.
We validate LEFT's performance on four different domains and seven tasks, ranging from 2D question answering, 3D referring expression comprehension, temporal sequence reasoning, and robotic manipulation. A general LEFT model significantly outperforms prior task-specific monolithic methods in performance and in data efficiency settings, and yields comparable performance to prior neuro-symbolic methods while requiring no pre-defined program implementations for each domain. Importantly, the unified LEFT framework can perform concept learning across domains, as well as show zero-shot transfer to three challenging and unseen reasoning tasks. In contrast, inference only LLM-based methods and general vision-language models fail to generalize.
LLM-based decomposition frameworks. Our framework integrates neural networks, logic reasoning, and large language models for commonsense reasoning. The first group of related literature studies LLM-based approaches. Inspired by the success of LLMs [Brown et al., 2020], many recent works have proposed frameworks that leverage LLMs to decompose text-based tasks into a sequence of API calls to existing models [Cheng et al., 2023, Dohan et al., 2022, Beurer-Kellner et al., 2023, Zelikman et al., 2023]. Compared to LEFT, these methods run inference only with LLMs and API models, and are limited to the language domain, without learning grounding to modalities. For example, LLMs may be able to reason about categories of objects inferred from language, but it
cannot recognize the candidate objects from the current scenes, or generate robotics movements to move the object. Works such as Gupta and Kembhavi [2023] and Surís et al. [2023] execute programs on images, but assume API access to a set of available modules with no further training. By contrast, LEFT learns to ground modular concepts across different domains by training concept embeddings for each modality, not requiring any pre-defined and pre-trained modules.
General vision-language models. In recent years, unified vision-language models (VLMs) have demonstrated success in learning on multimodal domains [Cho et al., 2021, Tsimpoukelli et al., 2021, Wang et al., 2022, Alayrac et al., 2022]. As a representative example, Flamingo [Alayrac et al., 2022] leverages powerful pre-trained vision and language models to achieve few-shot learning on language generation tasks, interleaving images, videos, and text, then outputting associated texts. However, these unified VLMs, and broadly end-to-end methods, are still limited in its application in different domains. In addition, in practice, they struggle with more complex unseen tasks. By contrast, LEFT performs well across domains, and can generalize to new challenging tasks that it has never seen.
Neuro-symbolic methods. Neuro-symbolic approaches that combine programs with neural networks have shown success in a variety of visual reasoning domains, with strong data efficiency and generalization abilities from their modular designs. Neuro-symbolic VQA [Yi et al., 2018] proposed symbolic program execution for the question-answering task, and the Neuro-Symbolic Concept Learner [NSCL; Mao et al., 2019] further improved the training paradigm by removing the requirement for dense supervision. Inspired by the success in 2D image domains [Andreas et al., 2016, Johnson et al., 2017b, Han et al., 2019, Hudson and Manning, 2019, Li et al., 2020, Chen et al., 2021a, Mao et al., 2021], neuro-symbolic methods have been introduced for grounding in 3D scenes [Achlioptas et al., 2020, Hong et al., 2022, Hsu et al., 2023], reasoning in temporal sequences [Chen et al., 2021b, Endo et al., 2023], and robotic manipulation [Wang et al., 2023, Mao et al., 2022]. However, these neuro-symbolic works require pre-defined domain-specific languages, such that task instructions are parsed into a constrained set of programs, and each program is then manually implemented in code. Therefore, it is challenging to incorporate commonsense reasoning into them, or generalize to different domains. In contrast, LEFT retains all the benefits of neuro-symbolic learning, while proposing a universal language for all domains. LEFT requires neither any domain-specific definition nor domain-specific program examples. It only leverages structures encoded through minimal prompting examples of first-order logic usage for the LLM; hence, it is generally applicable across domain.
3 Logic-Enhanced Foundation Model (LEFT)
Our Logic-Enhanced Foundation Model (LEFT) is a unified framework for concept learning and reasoning across different domains and tasks. It integrates large language models with differentiable logic modules, and modular neural networks for grounding concepts in each modality. Shown in Fig. 2, our system consists of three main components:
1. The first is a domain-independent LLM language interpreter, which generates first-order logic queries to the FOL execution engine (Section 3.1). The generated symbolic programs are represented with a hierarchical first-order logic structure.
2. The second is a domain-independent FOL executor, which executes the logic programs differentiably based on features of entities in the domain: objects, relations, actions, etc. (Section 3.2). LEFT's executor is fully differentiable, which allows for backpropagation.
3. The third is the domain-specific grounding modules, which consist of encoders for each modality that extract entity-centric and relational features, as well as the corresponding concept embeddings implemented as modular neural networks (Section 3.3). The concept grounding modules are initialized with concepts generated by the LLM interpreter.
LEFT is a framework of domain-independent reasoning modules (composed of the LLM interpreter and the FOL executor) and domain-specific grounding modules. With these components, LEFT conducts concept learning and reasoning across domains and tasks.
3.1 Domain-independent LLM interpreter
We leverage a pre-trained large language model as our domain-independent language interpreter. It takes language queries as inputs and generates first-order logic queries to downstream modules. The
Domain-independent
Domain-specific
Grounding Modules
Figure 2: At the core of LEFT is a foundation model-powered language interpreter that generates first-order logic (FOL) programs from language queries, a domain-independent, differentiable FOL executor that operates on all types of entities, and domain-specific grounding modules that connect symbols in the first-order logic language with domain-specific features.
examples we used to prompt the LLMs is a minimal set of general, domain-independent examples and system-level description of syntax rules in our FOL language. The LLM handles both natural language understanding and commonsense reasoning. For example, it needs to resolve coreferences and ambiguous modifier attachments: "Look at the book on the shelf, the blue one, select the object left of it." Furthermore, it performs commonsense reasoning, such as "Find an object that can fit into a row of blue spheres." The query it generates for the downstream FOL reasoning module will be non-ambiguous programs:  $\text{ix.blue}(x) \wedge \text{sphere}(x)$ . This strategy maximally harnesses the reasoning capability of LLMs across tasks, while retaining strong generalization. We show in experiments that the LEFT interpreter can reason about analogies, resolve coreferences, and tackle complex puzzles.
Formally, given an input task query  $Q$  in language, our LLM interpreter outputs programs  $P$ , which are programs written in first-order logic that will be passed to downstream FOL executors. Each program is composed of a function name and arguments, and can be chained together hierarchically such that the output of one function can be the input argument to another function. The LLM-interpreted program contains domain-independent FOL functions, which can take outputs from domain-specific functions as arguments. The FOL functions are a general set of pre-defined programs with built-in implementations and are listed in full in Table 1. These functions are either logic and numeric operations (such as counting, forall, exists), or functions that handle inputs and outputs (e.g., return a text description of object categories, or execute an action output by other modules). We present the syntax and built-in components of our FOL language in the next section.
LEFT leverages GPT-3.5 [Brown et al., 2020] as its LLM backbone. Importantly, the LEFT prompts are domain-independent, including simple syntax rules and examples written with general concepts such as cat and apple that do not occur in the any of our evaluation domains. Examples in our prompt also have minimal complexity, such as first-order logic sentences for "Is there an apple next to the cake", paired with short descriptions of FOL usage, such as "to classify whether two objects have a certain property, use 'on(x, y)'." Additionally, we use a step-by-step prompting paradigm, which
Name	Syntax	Description
variables	x,y,z,···	Variables that refer to entities in the domain.
∃, ∀	exists(var, expr)	expr is an expression that contains the variable var; return true if there is at least one assignment of var that satisfies expr.
iota(ι)	iota(var, expr)	expr is an expression that contains the variable var; return an assignment to var's that satisfies expr.
not	not expr	Compute the negation of an expression.
and, or	expr1 and expr2	Compute the conjunction/disjunction of an expression.
count	count(var, expr)	Count the number of assignments to var that will make expr true.
=, <, >	eq(expr1, expr2)	Built-in number comparison functions.
view	view expr)	expr is an object, e.g., view(ιx.blue(x)); it changes the view direction to face the object.
describe	describe expr)	expr is a text description, e.g., describe(ιc.color(c,ιx.sphere(x))); it re-turns the description (the color of the sphere).
do	do expr)	expr is an action, e.g., do(ιa_pick-up(a,ιx.sphere(x))); it executes the action (pick up the sphere).
concepts	left(x,y)	A function that takes objects, actions, and texts as inputs and returns Boolean values.
Table 1: LEFT's first-order logic language, a general reasoning language across domains.
asks the LLM to first reason about the query and simplify the text in language form, then generate FOL programs.
3.2 Domain-independent first-order logic executor
The LEFT domain-independent executor operates on first-order logic, which serves as a general reasoning language shared across different tasks. Given the parsed FOL programs  $P$  from the LLM interpreter, the executor operates  $P$  with grounding modules to return the final answer. The programs are executed recursively to model the hierarchical structure of the FOL reasoning process. The LEFT programs are implemented as differentiable first-order logic modules in Python code.
Our FOL language contains three parts: first, basic first-order logic operations such as Boolean operations and logic quantifiers; second, built-in functions such as describe, which generate a natural language response for an entity or a pair of entities, and do, which execute a robotic action; and third, domain-specific concept names, including object properties (e.g., cat), object relations (e.g., left), and actions (e.g., pack). These concept predicates are automatically produced by the LLM based on the language query, instead of being selected from a pre-defined set of functions; hence, the expressiveness of LEFT is not limited by the given set of primitives such as exists and count. It can learn to execute any new functions proposed by the LLM.
For a given universe of entities, the LEFT executor executes all expressions using a tensor-based representation of truth values. For example, given a scene with  $N$  objects, the execution result of the expression sphere(x), where  $x$  is an unquantified variable, can be represented as a vector of length  $N$ , where each entry  $i$  denote whether object  $i$  is a sphere. Relation expressions such as left(x,y) can be represented as matrices of size  $N \times N$ . Whether an object has been selected can be represented similarly, as a vector of size  $N$ , where each entry  $i$  is the probability of object  $i$  being selected. In practice, values such as sphere(x) and left(x,y) will be predicted by domain-specific grounding neural networks. In Table 1, we detail the execution strategy for all operations. Essential to our goal of a general neuro-symbolic framework, LEFT only requires these listed FOL functions to reason across numerous domains, including 2D, 3D, temporal, and robotic manipulation.
The programs generated from our LLM interpreter include FOL programs along with domain-specific concepts, which are executed by our grounding modules. The LEFT executor implements these programs based on the arity (e.g., number of arguments in a function) of the first-order logic program. unary-level functions are implemented with entity-centric features of size  $N$  for each entity from the input, and binary-level functions are implemented with relational features of size  $N \times N$ , etc. Notably, each domain-specific concept is initialized by the LLM, and the function implementation is executed with modular neural networks, which does not require manual definition in code.
Execution. We describe implementations for the FOL programs below. See Appendix for details.
exists(var, expr): We first recursively execute expr and get a tensor where var is one of the dimensions of the tensor. We perform a max pooling over that dimension. For example, the execution of sphere(x) yields a vector of length  $N$ , for exists(x, sphere(x)), we take the max value of the returned tensor.
iota(var, expr): We recursively execute expr and get a vector where var corresponds to the dimensions of the vector. We then perform a softmax operation over the vector.
and  $(\text{expr}_1, \text{expr}_2)$ : The recursive execution of  $\text{expr}_1$  and  $\text{expr}_2$  will yield two tensors. The conjunction of two resulting tensors is done by taking the element-wise minimum operation for two tensors. The same execution strategy is used for or operations (by taking the element-wise maximum) and logical negation (by taking the element-wise negation).
count(var, expr): We recursively execute expr, then perform a sigmoid operation followed by a sum over the vector to compute the expected number of objects.
$\mathbf{eq}(\mathbf{expr}_1, \mathbf{expr}_2)$ : The recursive execution of  $\mathbf{expr}_1$  and  $\mathbf{expr}_2$  will yield two scalar values  $s_1$  and  $s_2$ . We define the probability of  $s_1 = s_2$  as  $\sigma(\alpha \cdot (\gamma - |s_1 - s_2|))$ , where  $\alpha = 8$  and  $\gamma = 0.25$  are fixed hyperparameters. Less than and greater than operations are implemented similarly.
left(x, y): As an example of a domain-specific concept, the left function takes two variables as its input, and generates a score matrix of size  $N \times N$ . This is done by applying MLPleft on the binary embedding extracted by the feature extractor. Note that throughout the execution, we keep the "logit" of the predicted values instead of normalizing them to [0, 1] with a sigmoidal activation. This improves the numerical stability of the execution.
3.3 Domain-specific grounding modules
The final component of LEFT is the domain-specific grounding modules that connect concepts in the first-order logic language with modality-specific representations. LEFT does not depend on a pre-defined set of lexical concepts, and relies on LLMs to automatically extract concept names from language in the data. Our framework then initializes domain-specific grounding modules accordingly to learn the grounding of these concepts from training data. In other words, the grounding modules do not require manual definitions, can support execution of any given query, and are realized with a generic mechanism described below. The grounding modules are implemented with modular neural networks, consisting of modality-specific encoders and concept embeddings. We first extract entity-centric and relational features with modality-specific encoder  $\varepsilon$ . For 2D images,  $\varepsilon$  is Faster R-CNN [Ren et al., 2015] for encoding object-centric features and DenseCLIP [Rao et al., 2022] for generating object attention masks. For 3D point clouds,  $\varepsilon$  is PointNet++ [Qi et al., 2017]; and for motion sequences,  $\varepsilon$  is the Two-Stream Adaptive Graph Convolutional Network [Shi et al., 2019]. Unary features are represented as a set of features for each entity, binary features as a 2D matrix of relations between each pair of entities, and ternary relations as a 3D matrix, etc.
Given entity-centric representations, we then execute concept functions such as sphere and left(x, y). In general, we can have arbitrary neural networks that take the relational representation of entities and outputs a (soft) Boolean value from [0, 1]. In this paper, we initialize a multi-layer perceptron (MLP) for each concept that has been mentioned by the large language model, to represent the concept embedding. For example, in our 2D image domain, each object  $i$  has a vector embedding  $obj_{i}$  extracted by a Faster R-CNN encoder. We apply MLP_sphere on each object representation  $obj_{i}$  and generate a vector of length  $N$ , which is the execution result for the expression  $sphere(x)$ .
The grounding modules are automatically initialized based on the concept name as well as arity proposed by the LLM. For example, if a concept is unary (e.g.,  $red(x)$ ), the grounding module will be a MLP layer that maps object representations to the aforementioned soft Boolean value scalar. For the concept beside, the LLM specifies that the concept takes in features of two entities, and hence LEFT creates an MLP that operates on binary features. The domain-specific feature extractors and concept MLPs are trained through backpropagation, as our executor is fully differentiable.
We evaluate LEFT on four domains and seven tasks, and show its effectiveness in multiple settings. In Section 4.1, we compare LEFT to prior state-of-the-art neuro-symbolic and monolithic end-to-end
2D question answering
Query: Are there two objects of the same color nearly touching?
Figure 3: LEFT is evaluated on four broad categories of tasks, ranging from concept learning in 2D images, 3D point clouds, temporal motion sequences, and robotic manipulation.
3D referring expression
Query: Choose the ottoman sitting next to the end table.
Query: What body part does the person use before they turn?
Temporal sequence reasoning
Robotic manipulation
Query: Pack the ring into the brown box.