What’s Left? Concept Grounding with Logic-Enhanced Foundation Models
Authors: Joy Hsu
Jiayuan Mao
Joshua B. Tenenbaum
Jiajun Wu
Venue: NeurIPS 2023 poster (2023)
Keywords: concept learning
visual reasoning
large language models
neuro-symbolic learning
TL;DR: A unified concept learning and reasoning framework that integrates LLMs with learnable grounding modules across domains.

Abstract: Recent works such as VisProg and ViperGPT have smartly composed foundation models for visual reasoning—using large language models (LLMs) to produce programs that can be executed by pre-trained vision-language models. However, they operate in limited domains, such as 2D images, not fully exploiting the generalization of language: abstract concepts like “*left*” can also be grounded in 3D, temporal, and action data, as in moving to your *left*. This limited generalization stems from these inference-only methods’ inability to learn or adapt pre-trained models to a new domain. We propose the **L**ogic-**E**nhanced **F**ounda**T**ion Model (**LEFT**), a unified framework that *learns* to ground and reason with concepts across domains with a differentiable, domain-independent, first-order logic-based program executor. LEFT has an LLM interpreter that outputs a program represented in a general, logic-based reasoning language, which is shared across all domains and tasks. LEFT’s executor then executes the program with trainable domain-specific grounding modules. We show that LEFT flexibly learns concepts in four domains: 2D images, 3D scenes, human motions, and robotic manipulation. It exhibits strong reasoning ability in a wide variety of tasks, including those that are complex and not seen during training, and can be easily applied to new domains.

What's Left? with Logic-Enhanced Foundation Models
Concept Grounding
Joy Hsu"
Jiayuan Mao" MIT
Joshua Tenenbaum
Jiajun Wu Stanford University
Stanford University
MIT
Abstract
Recent works such as VisProg and ViperGPT have smartly composed foundation models for visual reasoning-using large language models (LLMs) to produce programs that can be executed by pre-trained vision-language models. However, fully exploiting the generalization of language: abstract concepts like "left" can also be grounded in 3D, temporal, and action data, as in moving to your left. This limited generalization stems from these inference-only methods' inability to learn or adapt pre- pre-trained models new domain. We propose the Logic-Entee Enhinned FoundaTion Model (LEFT), unified framework that learns to ground and reason with concepts across domains with a differentiable, domain-independdnt, first-order logic-based program executor. LEFT has an L.LM interpreter that outputs program represented in a general, logic-basee reasoning language, which shared across all domains and tasks. LEFT's executor then executes the program with trainable domain- specific grounding modules. We show that LEFT flexibly learns concepts in four domains: 2D images, 3D scenes, human motions, and robotic manipulation. It exhibits strong reasoning ability in a wide variety of tasks, including those that are complex and not seen during training, and can be easily applied to new domains.
they operate in limited domains, such as 2D images,
R
left
Query: there object to the left of the target?
(Dbjec,, lambda x:
LLM
left(x, iota(Object, lambda y: target (y)))
Figure 1: LEFT is unified concept learning and reasoning framework that grounds modular concepts across domains, and flexibly reasons with concepts across tasks with foundation model.
Introduction
The power of language lies in its generalization based on abstraction. A single concept, such as left, can be used across domains: perception, we recognize the left leg from a chair's 3D point cloud; in navigation, we go home by turning left around the cafe at the end of the block; in manipulation, we pick and place a mug on the left of a plate. Left has domain-specifes groundings each of these
Eqqual contribution. Email: joyc (eyyjjsstaoorddcco edu
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

domains, but the concept symbol itself as an abstraction serves as a basis for complex and multi-step reasoning across domains.
Machine systems for visual reasoning also henefit from the disentanglement of concept learning and reasoning. Most recently, models such as VisProg [Gupta and Kembhavi 2023) and ViperGPT (Surís et al. 2023] have leveraged large language models (LLMs) to produce programs based on language queries. For example. to answer "what's the color of the cat?", the program will first locate the cat, then query its color. Such programs are executed with pre-defined and pre-trained vision- language models (e.g., open-vocabulary segmentation models) for grounding and question- -ansiionn....ring.... These works perform well on image-based reasoning, though surprisingy, despite the generalization seemingly offered for free by language, less success has been shown other domains.
What's left to make visual reasoning systems work across domains? We identify two key shortcomings of existing methods. First, they are inference-only, building exclusively upon pre-trained models to interpret concepts such as left during program execution. Therefore, in domains where data is scarce (e.g.. 3D, human motion, or robot action), pretrained models, if there exist any, do not perform as well. Thus, we need trainable systems that learn the domain-speciffe groundings of concepts to execute across domains. Unfortunately, these visual reasoning models also cannot be made trainable due to their second shortcoming: they rely on a non-differentiable program executor to compose these grounding modules. Therefore, though left may have different groundings in 3D scenes and robot actions compared to 2D images, these modules will not be able to adapt through additional training those domains.
We address these issues by proposing Logic-Enhanced Foundation Models (LEFT), a model that learns ground and reason with concepts across domains. LEFT also leverages LLMs for language parsing: but unlike prior work, LEFT has trainable concept grounding modules that learn from domain data, made possible via differentiable, domain--nndependet,, first-order logic-based program execu- tor. Inspired by previous works that combine symbolic programs and deep models via differentiable execution [Mao et al. 2019 Dong et al. 2019]. our design no longer requires manually designed domain-spectie specific languages and facilitates generalization across domains. Specifically, LEFT's LLM interpreter takes as input language queries, resolves textual ambiguities, and outputs a -mbiguuus program represented in a general reasoning language of first-order logic, which is shared across domains and tasks. LEFT's executor then executes the logic program with learnable domain-specific grounding modules, which are automatically initialized with concepts generated by the LLM.
LEFT enjoys the advantages of strong performance and data efficiency across domains due to its modular structure. Within a single domain, LEFT generalizes zero-shot to unseen and complex tasks, by leveraging general reasoning capabilities from the LLM interpreter, and effectively recomposing learned, grounded concepts with the generic first-order logic executor. LEFT can be viewed as a generalized framework of VisProg and ViperGPT: in domains where pre-trained models are available and training is not required (c.g., 2D images), LEFT can similarly be used inference-only.
We validate LEFT's performance on four different domains and seven tasks, ranging from 2D question answering, 3D referring expression comprehension, temporal sequence reasoning, and robotic manipulation. A general LEFT model significantly outperforms prior task-specific monolithic methods in performance and in data efficiency settings, and yields comparable performance to prior neuro-symbolic methods while requiring no pre-defined program implementations for each domain. Importantly, the unified LEFT framework can perform concept learning across domains, as well as show zero-shot transfer three challenging and unseen reasoning tasks. In contrast, inference only LLM-based methods and general vision- languae models fail to generalize.
2 Related Work
LLM- decomposition frameworks. Our framework integrates neural networks, logic reason- ing, and large language models for commonsense reasoning. The first group of related literature studies LLM-based approaches. Inspired by the success of LLMs [Brown et al. 2020). J many recent works have proposed frameworks that leverage LLMs to decompose text- text-based tasks into a sequence
of API calls to existing models [Cheng et al. 2023 Dohan et al.) 2022 Beurer-Kellner et al., 2023 Zelikman et al. 2023]. Compared to LEFT, these methods run inference only with LLMs and API models, and are limited to the language domain, without learning grounding to modalities. For example, LLMs may be able to reason about categories of objects inferred from language, but
2

---

cannot recognize the candidate objects from the current scenes, or generate robotics movements to move the object. Works such as Gupta and Kembhavi [023] and[Su al. 2023] execute programs on images, but assume API access to a set of available modules with no further training. By contrast, LEFT learns to ground modular concepts across different domains by training concept embeddings for each modality, not requiring any pre-defined and pre-trained modules.
General vision-language models. In recent years, unified vision-langune models (VLMs) have demonstrated success in learning on multimodal domains [Cho et al. 2021 Tsimpoukelli et al. 2021
Wang al. 2022 Alayrac et al. 2022]. As representative example, Flamingo [Alayrac al. 2022
leverages powerful pre-trained vision and language models to achieve few-shot learning on language generation tasks, interleaving images, videos, and text, then outputting associated texts. However, these unified VLMs, and broadly end-to-end methods, are still limited in its application in different domains. In addition, in practice, they struggle with more complex unseen tasks. By contrast, LEFFT performs well across domains, and can generalize to new challenging tasks that it has never seen.
Neuro-symbolic methods. Neuro-symbolic approaches that combine programs with neural net- works have shown success in variety of visual reasoning domains, with strong data efficiency and generalization abilities from their modular designs. Neuro-symbolii VQA (Yi et al., 2018 proposed symbolic program execution for the question-annsering task, and the Neuro-Symbolic Concept Learner [NSCL; Mao et al., 2019] further improved the training paradigm by removing the requirement for dense supervision. Inspired by the success in 2D image domains [Andreas
al. 2016 Johnson et al.
[2017b Han et al., 2019 [udson and Manning 2019 Li et al. [020
Chen et al. 2021a Mao et al. [022] neuro-symbolic methods have been introduced for grounding
3D scenes Achlioptas et al. 2020 Hong al. 2022] Hsu et al. 2023]. reasoning temporal sequences [Chen et al. 2001bb) Endo et al. [023]] and robotic manipulation [Wang et al.. 2023 Mao
(ct al. [022]. However, these neuro-symbolic works require pre-defined domain-specifie languages, such that task instructions are parsed into a constrained set of programs, and each program is then manually implemented in code. Therefore, is challenging to incorporate commonsense reasoning into them, generalize to different domains. In contrast, LEFT retains all the benefits of neuro- symbolic learning, while proposing a universal language for all domains. LEFT requires neither any comain-speciffe definition nor domain-specinf program examples. It only leverages structures encoded through minimal prompting examples of first-order logic usage for the LLM; hence, is generally applicable across domain.
3 Logic-Enhanced Foundation Model (LEFT)
Our Logic-Enhanen Foundation Model (LEFT) is a unified framework for concept learning and reasoning across different domains and tasks. It integrates large language models with differentiable logic modules, and modular neural networks for grounding concepts in each modality. Shown in Fig. 2 our system consists of three main components:
1. The first is domain-independenn LLM language interpreter, which generates first-order logic queries to the FOL execution engine (Section The generated symbolic programs are represented with hierarchical first-order logic structure.
2. The second is a domain- conaen-indeeendent FOL executor, which executes the logic programs differentiably based on features of entities in the domain: objects, relations, actions, etc. (Section 3.2). LEFT's executor is fully differentiable, which allows for backpropagation.
3. The third is the domain-specifi grounding modules, which consist of encoders for each modality that extract entity-centee and relational features, as well as the corresponding concept embeddings implemented as modular neural networks (Section 3.3). The concept grounding modules are initialized with concepts generated by the LLM interpreter.
LEFT is framework of domain-independdnt reasoning modules (composed of the LLM interpreter and the FOL. executor) and domain-specific grounding modules. With these components, LEFT conducts concept learning and reasoning across domains and tasks.
3.1 Domain-independent LLM interpreter
We leverage a pre-trained large language model as our domain-independdnt language interpreter: takes language queries inputs and generates first-order logic queries to downstream modules. The
3

---

Domsin-independent
LLM Interpreter
step-by-step prompting paradigm, which First-Order Logic Executor
Query: How many objects are left sphere?
forall expr)
exists(var, expr)
softmax -0.1, 1.8])
LLM
iota Lvar,var, expr)
[0.2, 8.1, 0.71
not nexpr))))
count(Doject,
and and expr2)
left(x, (Object, lambda y: sphere(y))))
or(expr1, expr2)
Sum(signoid((B.B, 1.4, 2.1]))
count(var, expr)
2.. 3838
count
<,x,p1 expr2)
left
sphere(x)
8.3, -1.3, 2.7
x
iota
left(x, y) + MLP1ett
8.8, 8.4, 1.2
-2.5, 0.2, 0.6
sphere
Domain-specific
Grounding Modules
Modality input
Relational features
Faster R-CNN
unary
Psphere
[0.5, -0.1, 1.8]
binary
PointNet++
0.3, -1.3, 2.7
MLP1sft
0.8, 0.4, 1.2 -2.5, 0.6
PATRRE
2s-AGCN
ternary
RPASR
MLPcenter
Dense CLIP
Figure 2: At the core of LEFT a foundation model-powered language interpreter that generates first-order logic (FOL) programs from language queries, domain-independent, differentiable FOL executor that operates on all types of entities, and domain-specific grounding modules that connect symbols in the first-order logic language with domain-specifie features.
examples we used to prompt the LLMs is a minimal set of general, Nomaiininndependee examples and system-level description of syntax rules in our FOL language. The LLM handles both natural language understanding and commonsense reasoning. For example, it needs to resolve coreferences and ambiguous modifier attachments: "Look at the book on the shelf, blue one, select the object left of it" Furthermore, it performs commonsense reasoning, such as "Find an object that can fit into a row of blue spheres." " 'The query it generates for downstream FOL reasoning module will be con-ambiguous programs: blue(x)) sphere(z). This strategy maximally harnesses the reasoning capability of LLMs across tasks, while retaining strong generalization. We show in experiments that the LEFT interpreter can reason about analogies, resolve coreferences, and tackle complex puzzles.
Formally, given an input task query Q in language, our LLM interpreter outputs programs P. which are programs written in first-order logic that will be passed to downstream FOL executors. Each program is composed of function name and arguments, and can be chained together hierarchically such that the output of one function can be the input argument to another function. The LLM- interpreted program contains domain-independent FOL. functions, which can take outputs from domain-specific functions as arguments. The FOL. functions are general set of pre-defined programs with built-in implementations and are listed full in Table [ These functions are either logic and numeric operations (such counting, forall, exists), or functions that handle inputs and outputs (c.g.. return a text description object categories, or exccute an action output by other modules). We present the syntax and built-in components of our FOL language in the next section.
LEFT leverages GPT-3.5 [Brown al., (2020) as LLM backbone. Importantly, the LEFT prompts are bommai-innependdeee such as cal and apple that do not oceur the any our evaluation domains. Examples in our prompt also have minimal complexity, such as first-order logic sentences for "Is there apple next to the cake", paired with short descriptions of FOL. usage, such as "t0 classify whether two objects have certain property, 'on(x. y)"." Additionally,
including simple syntax rules and examples written with general concepts

---

Name
Syntax
Description
variables
2,
Variables that refer to entities in the domain.
V
exists(var, expr)
expr expression that contains the variable var; return true if there is least one assignment var that satisfies expr. expr is an expression that contains the variable var; return an assignment to var's that satisfies expr.
iota (1)
iota(var, expr)
not
not expr
Compute the negation of expression.
and, or
expr, and expr count(var, expr)
Compute the conjunctiin/dijjunction of expression. Count the number assignments to var that will make expr true. Built-in number comparison functions. expr is an object, e.g., view(uæ (lue(blll))))))))) it changes the view direction to face the object.
count
<,>
eq(expr, expr.) view(expr)
view
describe
(excribe(expr)
expr a text description, e.g., describel (1 (e.coolor(e, (2 (x.sphere(=)))) re- turns the deseription (the color of the sphere). (-.spheee())) it executes the
do
do(expr)
expr is an action, e.g., action (pick up the sphere).
concepts
left(z, v)
function that takes objects, actions, and texts as inputs and returns Boolean values.
Table I: LEFT's first-order logic language, a general reasoning language across domains.
asks the LLM to first reason about the query and simplify the text in language form, then generate FOL programs.
3.2 Domain-independent first-order logic executor
The LEFT domain-independdnt executor operates on first-order logic, which serves as a general reasoning language shared across different tasks. Given the parsed FOL programs P from the LLM interpreter, the executor operates P with grounding modules to return the final answer. The programs are executed recursively to model the hierarchical structure of the FOL reasoning process. The LEFT programs are implemented as differentiable first-order logic modules in Python code.
Our FOL language contains three parts: first, basic first-order logic operations such as Boolean operations and logic quantifiers; second, built-in functions such as describe, which generate a natural language response for an entity or pair of entities, and do, which execute a robotic action; and third, domain- sppain- concept names, including object properties (e-g., car), object relations (c.g.. leff), and actions (c.g., pack). These concept predicates are automatically produced by the LLM based on the language query, instead of being selected from pre-defined set of functions; hence, the expressiveness of LEFT is not limited by the given set of primitives such exists and count. It can learn to execute any new functions proposed by the L.M..
For given universe of entities, the LEFT executor executes all expressions using a tensor-based representation of truth values. For example, given scene with N objects, the execution result of the expression sphere(z), where z is an unquantified variable, can be represented as vector of length N, where each entry denote whether object is sphere. Relation expressions such lef(xx. y) can be represented as matrices of size N x N. Whether an object has been selected can be represented similarly, as a vector of size N, where each entry the probability of object being selected. In practice, values such sphere(x:) and left(r, (x. will be predicted by domain-specinf grounding neural networks. In Table [ we detail the execution strategy for all operations. Essential to our goal of a general ceuro-symbolic framework, LEFT only requires these listed FOL functions to reason across numerous domains, including 2D, 3D, temporal, and robotic manipulation.
The programs generated from our LLM interpreter include FOL programs along with domain-specific concepts, which are executed by our grounding modules. The LEFT executor implements these programs based on the arity (e.g., number of arguments in a function) of the first-order logic program. Unary-level functions are implemented with entity-centric features of size N for each entity from the input, and binary-level functions are implemented with relational features of size N x N, etc. Notably, each domain-specific concept initialized by the LLM,, and the function implementation executed with modular neural networks, which does not require manual definition in code.
5

---

Execution. We describe implementations for the FOL programs below. See Appendix for details.
exists(var, expr): We first recursively execute expr and get a tensor where is one of the dimensions of the tensor: We perform a max pooling over that dimension. For example, the execution of sphere (2) yields vector of length N, for (pperr
take the max value of the returned tensor.
iota(var, expr): We recursively execute expr and get a vector where var corresponds to the dimensions of the vector. We then perform softmax operation over the vector.
and(expr,, expr2): The recursive execution of expr, and expr, will yield two tensors. The conjunction of two resulting tensors is done by taking the element-wis wise minimum operation for two tensors. The same execution strategy is used for or operations (by taking the element-wiss maximum) and logical negation (by taking the element-wise negation).
count var, expr): We recursively execute expr, then perform sigmoid operation followed by sum over the vector to compute the expected number of objects.
eq(expr, expry): The recursive execution of expr, and expry will yield two scalar values 81 and sg. We define the probability s1 = 82 as a (a (7 - |s| - =2))), where a = 8 and y = 0.25 are fixed hyperparameters. Less than and greater than operations
implemented similarly.
left(z, y): As an example of a domain- speciffc concept, the left function takes two variables as its input, and generates a score matrix of size N x N. This is done by applying MLP on the binary embedding extracted by the feature extractor. Note that throughout the execution, we keep the "logit" of the predicted values instead of normalizing them to [0, 1] with sigmoidal activation. This improves the numerical stability of the execution.
3.3 Domain-specific grounding modules
The final component of LEFT is the domain-specific grounding modules that connect concepts in the first-order logic language with modality-specifit representations. LEFT does not depend on pre-defined set of lexical concepts, and relies on LLMs to automatically extract concept names from language in the data. Our framework then initializes domain-speciffi grounding modules accordingly to learn the grounding of these concepts from training data. In other words, the grounding modules do not require manual definitions, can support execution of any given query, and are realized with a generic mechanism described below. The grounding modules are implemented with modular neural networks, consisting of modality-speciffi encoders and concept embeddings. We first extract entity-centric and relational features with modality-speciffe encoder For 2D images, is Faster R-CNN [Ren et al. 2015] for encoding object-centric features and DenseCIIIP [Rao et al. 2022 for generating object attention masks. For 3D point clouds, e is PointNet++ (Qi et al. 2017 |; and for motion sequences, 6 is the Two-Stream Adaptive Graph Convolutional Network (Shi ct al. 2019).
Unary features are represented as set of features for each entity, binary features as a 2D matrix of relations between each pair of entities, and ternary relations as a 3D matrix, etc.
Given entity-centric representations, we then execute concept functions such as sphere and left(x, y). In general, we can have arbitrary neural networks that take the relational representation of entities and outputs a (soft) Boolean value from [0, 1]. In this paper, we initialize multi-layer perceptron (MIP) for each concept that has been mentioned by the large language model, to represent the concept embedding. For example, in our 2D image domain, each object i has vector embedding obj, extracted by a Faster R-CNN encoder. We apply MLP ML.ppphere sphere on each object representation obj; and generate vector of length N. which is the execution result for the expression sphere(z).
The grounding modules are automatically initialized based on the concept name well as arity proposed by the LLM. For example, if concept is unary (e.g., red(x)), the grounding module will be a MLP layer that maps object representations the aforementioned soft Boolean value scalar. For the concept beside, the LLM specifies that the concept takes in features of two entities, and hence LEFT creates an MLP that operates on binary features. The domain- speccifc feature extractors and concept MI.Ps are trained through backpropagation, our executor is fully differentiable.
Experiments
We evaluate LEFT on four domains and seven tasks, and show its effectiveness in multiple settings. In Section 4.1 compare LEFT to prior state-of-ther of-the-rtttttttt neuro-symbolic and monolithic end-to-end

---

2D question answering
3D referring expression
Temporal sequence reasoning
Robotic manipulation
Query: Are there two objects of same color nearly touching?
Query: Choose the ottoman sitting next to the end table.
Query: What body part does the penson use before they tum?
Query: Pack the ring into the brown box.
Figure 3: LEFT is evaluated on four broad categories of tasks, ranging from concept learning in 2D images, 3D point clouds, temporal motion sequences, and robotic manipulation.
methods concept learning across four domains. Our model excels at task performance and also data efficiency, without requiring any domain-speciie knowledge, Next, in Section (.2]. we present LEFT's zero-shot generalization performance to three novel and challenging visual reasoning tasks, and compare its reasoning capabilities to that of L.LM-based decomposition frameworks and general vision-language model. Implementation and training details for LEFT be found in the Appendix.
4.1 Concept learning across domains
We train and validate LEFT on six different datasets, grouped into four domains (Figure 3)
2D visual concept learning. the first domain, we focus on the task of visual question answering in 2D images: given an image and a language query (c.g., "How many small cylinders are partially hidden?" ")). the goal to predict an answer from set vocabulary. The accuracy is calculated as the exact match to ground truth answers. We validate LEFTT on the CLEVR and CLEVR R-umanns dataset (Johnson et al.) 2017a], which contain images with objects rendered of different shape, size, material, and color. CLEVR contains synthetically generated queries, while the ELEVR-Humaan is human-annotated dataset with diverse natural language questions.
3D visual concept learning. The second domain evaluates 3D referring expression comprehension. The input is a set of object point clouds in the scene, and a query (c.g. "The monitor that you would class in the middle of the other two") Models are evaluated by the accuracy of selecting the correct target object. In the 3D domain where we need reference frames to disambiguate relations like left, the grounding of such concepts will consider the viewpoint specified in the language query, through a generic high-arity function that does not require a specific implementation. We evaluate LEFT on the ReferIt3D dataset [Achlioptas et al., 2020], with 3D point clouds from ScanNet [Dai] et al. 2017). a collection of indoor scenes. ReferIt3D consists of two subsets, SR3D, which focuses on spatial reference with synthetically generated queries, and NR3D, with human-annotated queries. We use the NR3D subset from NS3D, which contain the same set of concepts as in SR3D.
Motion capture concept learning. The third domain contains human motion capture. In particular, we focus on the task of question answering in temporal motion sequences. Given a sequence of human motion, parameterized by joints, and question (e.g." "What action does the person do before they move backward and after they kick?"), the goal is to predict the exact match answer. We train and evaluate LEFT on the Motionnt dataset [Endo et al. 2023). which asks questions about temporal relations and a variety of attributes, including action, direction, and body parts.
Robotic manipulation. Finally, we demonstrate that LEFT can learn action concepts that corre- spond to robotic motions, in the context of robotic manipulation. The input to this task is an image representing a tabletop environment, and a language instruction (e.g.. "Pack the yoshi figure into the brown box." ") The goal generate sequence of actions that maximizes the average total reward over this robotic manipulation task. We validate LEFT on the CLIPort dataset [Shridhar et al. 2022
focusing on the task of sequential packing Google Scanned Objects (Downs al. 2022
4.1.1
Accuracy
We compare LEFT against two categories of hop-erforming models across all six datasets. The first category contains neuro-symbolic models, which require a pre-defined domain-specific language and implementations for each program. The second category contains monolithic end-to-end methods which usually require a significant amount of data to train. Notably, our unified LEFT framework able perform all domains, instead of requiring specific reasoning modules for each setting.
7

---

Pre-def.
CLEVR
SR3D
HumanMotionQA
Cliport
Prior NS
Yes
0.992 (NSCL)
0.627 (NS3D)
0.578 (NSPose)
0.797 (ProgramPort)
And
No
0.989 (MAC)
0.670 (BUTD
0.430 (MotionCLIP)
0.708 (Cliport)
LEFT
No 0.996
0.620
0.582
0.793
Table 2: Results comparing LEFT to prior top- performing neuro-symbolic and end-to-end works. Our unified LEFT framework yields strong performance on all settings without any pre-defined programs.
Table summmaiiees the results of different models on four datasets that involve synthetically gener- ated language, and hence ground truth programs exist for neuro-symbolic methods and our approach. We see that across CLEVR, SR3D, HumanMotion QA,, and CLIPort, LEFT achieves comparable performance to prior seuro-symbolic works [Mao et al. 2019 Hsu al. 2023 Endo al. 2023 Wang et al., 2023), and outperforms end-to-end methods in CLEVR, HumanMotionOA, and CLI- Port [Hudson and Manning 2018 Jain al. 2022 Shridhar et al. 2022 Importantly, our LEFT
model applies a unified language interpretation and program execution framework for across all domains and tasks, while prior work neuro-symbolic learning requires domain-specific languages and reasoning modules for each dataset. This suggests that first-order logic language can serve as an effective and general language for representing language queries.
In Table 3 we examine LEFT on tasks with human-annotated natural language queries. The first number for LEFT represents accuracy out of all queries where the program generated by I.MM is executable, while the latter number represents accuracy out of all queries, considering the percentage of LLM outputs that are not executable. We note that MDETR (Kamath it can better capture the similarity between similar concepts such as front and in-front-of. In con- trast, the representations for different concepts in LEFT are independent. The prior neuro-symbolic method we compared against outperforms LEFT on NR3D, as it also leverages an LLM for language interpretation, but uses set of domain domain-speeific examples in prompts that are hand-crafted.
Pre-def.
ELEVR-Humane
NR3D
Prior NS
Yes
0.678 (NSVQA)
0.526 (NS3D)
End-to-End
No
0.817 (MDETR)
0.430 (MVT) 0.496 / 0.485
LEFT
No
0.788 / 0.698
Table 3: Comparisons on natural language tasks.
2021 outperforms LEFT CLEVR-Humane possibly because
4.1.2 Data efficiency
We evaluate LEFT on data efficiency settings for CLEVR [Johnson et al. 201 and SR3D [Achiiop
tas al., 2020]. LEFT significantly outperforms all end-to-end methods every data efficient
percentage point tested, and performs comparably to best-performing neuro-symbolic methods [Hsu
al.
2023 Jain al., 2022, Huang
2022 Yang et al. 202 Mao al. 2019 Mascharka
et al. 2018 Hudson and Manning 2018 (See Table 4 and Table 5). The percentages indicate the
percentage train data used for training compared the full dataset.
Pre-def.
0.5%
1.5%
2.5%
5%
10%
100%
NS3D Yes
0.426
0.520
0.556
0.591
0.600
0.627
BUTD-DETR
No 0.083
0.158
0.259
0.395
0.528
0.670
MVT
No
0.161
0.275
0.322
0.380
0.491
0.645
SAT
No 0.172
0.260
0.298
0.330
0.362
0.579
LEFT
No
0.410
0.521
0.565
0.579
0.591
0.620
Table 4: Results on SR3D in data efficient settings. LEFT performs comparably to prior neuro- symbolic works, and outperforms end-to-end works significantly.
Focusing on CLEVR as an ablation study, we note that NSCL outperforms LEFT on the 10% setting. We attribute this gap to the additional domain-speciin particular, NSCL assumes the existence of four attribute spaces (color, shape, material, and size), and each concept lies in exactly one attribute space. Furthermore, all concepts within the same attribute
inductive biases that NSCL leverages. In
BBTTD operates constrained setting, assuming 485 classes instead of the full 607 classes in Refertt3D.

---

space are mutually exclusive. Therefore, the inductive biases encoded by NSCL can be seen as combination of hierarchical program-based reasoning mechanism, and additional domain cemain-specifie priors on concepts. By contrast, LEFT only leverages the program-base program-based reasoning mechanism, which is general across many domains and tasks. comparison of our model with other baselines and NSCI., we show that this general program-based reasoning mechanism is powerful enough to explain most of the data efficiency improvements.
4.2 Reasoning generalization across tasks
Finally, we present LEFT's ability to zero-shot generalize to unseen tasks.
Pre-def.
10%
25%
50%
100%
NSCL
Yes
0.989
0.992
0.994
0.992
CLEVR reasoning tasks. We gen- erate three zero-shot transfer tasks based on the CLEVR datasets [John son et al. 2017a with 100 evaluation examples each. See Figure 4 for an
TbD-Net
Yes
0.541
0.560
0.912
0.991
MAC
No
0.673
0.940
0.974
0.989
LEFT
No
0.941
0.991
0.992
0.996
Table 5: Results on CLEVR in data efficient settings.
example of each task. The CLEVR- Ref task consists of CLEVR image and language queries, requiring LEFT to locate the exact target object. The CLEVR-Puzzle task consists of images paired with visual puzzle descriptions, which necessitates variable assignment puzzle solving. The CLEVR task is based on Raven's ' Progressive Matrices, which consists an image and a 3x3 grid describe by language. This task requires the LLM interpreter to reason about patterns in the grid by row and column to determine the attributes of the target object, and generate queries to the FOL executor to check if such an object exists in the scene. We include dataset details in the Appendix.
CLEVR-Ref
CLEVV
CLEVR-RPM
Query: There is a cylinder that is behind the eyun sphere, find the sphere that behind it.
Query: Can find four objects from large
Query: There are 9 objects, ordered in 3x3 grid: row col I is a green eube; row col 2 is green sphere, row col is green cylinder; row 2 col is brown cube; row 2 col 2 is a brown sphere; row col 3 is a brown cylinder: row 3 col a cyan cube; row col 2 is eyan sphere; I am missing one object row col Can you find an object in scene that fit there?
the image such that: object blue sphere, object 2 is a large blue sphere; object 3 is a blue sphere, object is a blue rubber object; objoct 2 is right of objoct 3; object 2 is right of object 4; object right of object 2.
LLM Interpreter
sphera(x) behind(xx, inta(Object,, laebda y: cylinder(y) and behind(yy fotat,e laebda 2: cyan(z) and sphere(e))))))
exists (object, lamoda x2: oxitss(sfect,, lambda (ohjee,,, larada x4: Large
exists(Object,
blue(xt) and sphere(x1) and 3arge(s2) and blue(x2) and sphere(x2) and blue(x33 and sphere(xa) and blue(x4) and rubber(x4) and right(x2, x3) and right(s2, x4) and right(is, and right(sd, right(x), }?))
FOL Executor
exists (Dhject, cylinder(k))
eyan(x)
Answer: True
Answer: Object index
Answer: False
Figure 4:
The three zero-shot generalization tasks. As seen in the CLEVR-Ref example, LEFT's fomain-independce modules resolve the language into first-order logic and generate the answer.
4.2.1 Accuracy
We train LEFT on the CLEVR question answering task, and report zero-shot transfer performance on the three unseen reasoning tasks. In T we see that LEFT achieves strong performance on all datasets. We compare LEFT performance with ground truth programs (GT programs) and with FOL programs from our LLM interpreter (LLM programs). This ablation demonstrates that there are only small drops in performance with LLM programs as compared to ground truth programs across all three complex reasoning tasks. LEFT can zero-shot transfer to novel visual reasoning tasks with LLM generated first-order logic, and effectively reuse concept embeddings, enabling flexible

---

CLEVR-Ref CLEVR-Puzzles CLEVR-RPM
LEFT GT programs
1.00
0.92
1.00
LEFT + LLM programs
0.94
0.75
0.87
VisProg Gupta and Kembhavi
2023
0.35
0.27
0.51
ViperGP [Suris et
2025
0.08
0.34
0.04
OpenFlamingo 4-shol Awadalla al. 2023
N/A
0.54
0.52
OpenFlamingo 8-shot
N/A
0.57
0.52
Table 6: Zero-shot generalization of LEFT to three unseen tasks; LEFT recomposes domain-
independent programs and learned concepts with the LLM interpreter for flexible transfer.
challenging task which contains complex Raven's 's Progressive Matrices [Barrett et al. 2018 descriptions that require the LLM interpreter to reason about patterns described by language. While the prompts given to the
generalization. Importantly,
that LEFT performs well on CLEVR-RPM.
LLM are simple, our interpreter can solve complex tasks.
As baselines, compare LEFT with LLM-based decomposition frameworks and general vision- language models. VisProg [Gupta and Kembhavi 2023] and ViperGPT [Surís et al. 2023 are inference only, integrated LLM with APIs frameworks, that has shown success on variety of visual reasoning tasks. In Table we see that LEFT significantly outperforms VisProg and ViperGPT on the CLEVR transfer tasks. The -Ls-based decomposition frameworks often fail to output executable code for complex queries,; in addition, the accuracy of execution by the APIs is low. For example, on the CLEVR-RPM task, only 0.40 of queries were executed successfully by ViperGPTT, with accuracy of 0.10 out of executable queries. As these methods execute programs with a constrained set of pre-defined functions, the accuracy for executing undefined relations, such to the right of, which is learned in LEFT, is low. In addition, for complex domains such as 3D and temporal human motion sequences, it unclear what pre-defined models should be used for these inference-only frameworks.
For comparison to general vision-language models, we evaluate OpenFlamingo [Awadalla et al. 2023), ). an open-sourced version of Flamingo [Alayrac et al., 2022] OpenFlamingo is VL model backed by a CLIP vision encoder [Radford et al. [202]] and a LLAMA language encoder [Touvron) al. 2023 On both 4- and 8-shot variants, LEFT outperforms OpenFlamingo significantly. Notably, OpenPlamingg Plmminoo cannot predict answers for CLEVR- Ref, which requires an object index as output.
Discussion
Foundation Model (LEFT), a unified concept learning and reasoning framework that flexibly learns concepts across different domains, and generalizes reasoning to unseen, complex tasks. Our framework combines foundation models and first-order logic executors as domain-iindeenddnt reasoning modules, and learnable grounding modules for specific domains. LEFT represents step towards modeling general reasoning capability in a rightfully grounded way.
We have presented the
LEFT leverages pre-trained LLMs for domain-indepenade system are limited by the quality of the LLM, which we cannot finetune. For syntax errors in LLM-generated programs, we can easily detect them and resample. However, currently cannot recover from semantic errors in the programs. A future direction would be to incorporate execution traces or failures as additional inputs to LLMs such that they fix the queries they generate.
reasoning. Hence, the results of our
LEFT may also fail with insufficient domain data, it cannot effectively train the domain domain-specific grounding modules. We note that this is limitation for most learning systems trained solely on one given dataset. To partially mitigate this issue, LEFT instructs LLMs to canonicalize concepts when reasonable, which helps learning certain rare concepts. It is also straightforward to incorporate externally trained modules, by directly integrating pretrained visual models as domain-specific grounding modules. For example, we can plug in Grounding DINO for object detection if we have prior task knowledge [Liu et al.) 2023]. However, for some domains, there are no off-the-shel visual recognition models (c.g., relations and human motions); therefore, LEFT can train its concept groundings by learning from data. Notably, LEFT is also more data efficient compared to end-to-end methods due to its modular structure, it can decompose complex concepts into primitive ones.
Our framework currently does not model the interaction between the language and perception. Instead, LEFT relies on context-dependent grounding modules which learn from data to resolve ambiguous language. Future directions for LEFT include building in pragmatic reasoning frameworks, modelling speaker intents, and leveraging feedback from perception execution.
10

---

Acknowledgments. We thank Weiyu Liu and Gabriel Poesia for providing valuable feedback on the paper. This work is part supported by the Stanford Institute for Human- Human-Centered Artificial Intelligence (HAI), ONR MURI N00014-22-1-27444 ONR N00014-23-1-2355. Air Force Office of Scientific Research (AFOSR) YIP FA9550-23-1-0127, the Center for Brain, Minds, and Machines (CBMM), the MIT Quest for Intelligence, MIT-IBM Watson AI Lab, Analog Devices, Autodesk, J.P. Morgan, and Salesforce. JH is supported by the Knight Hennessy Scholarship and the NSF Graduate Research Fellowship.
References
Panos Achlioptas, Ahmed Abdelreheem. Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Refertt3D; Neural Listeners for -rnenGr 3D Object Identification in worl-World Scenes. In ECCV, 2020.
Lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: Visual Language Model for Few-Shot Learning. In NeurIPS, 2022.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Leamnngg Compose Neural Networks for Question Answering. NAACL-HLT, 2016.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. OpenFlamingo: An opennsouuree Framework for Training Large Autoregressive Vision-language Models. arXiv preprint arXiv:2308. 01 390, 2023.
David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Mcasuring Abstract Reasoning in Neural Networks. In ICML, 2018.
Luca Beurer-Kellner, Mare Fischer, and Martin Vechev. Prompting Is Programming: A Query Language for Large Language Models. In PLDI, 2023.
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language Models are Few-Shot Learners. In NewrlPS, 2020.
Wenhu Chen, Zhe Gan, Linjie Li, Yu Cheng, William Wang, and Jingjing Liu. Meta Module Network for Compositional Visual Reasoning, In WACV, 2021a.
Zhenfang Chen. Jiayuan Mao, Jiajun Wu, Kwan-Ya Yee Kenneth Wong. Joshua B Tenenbaum, and Chuang Gan. Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning. ICI.R, 2021b.
Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, et al. Binding Language Models in Symbolic Languages. In ICLR, 2023.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying Vision and Language Tasks via Text Generation. In ICMI,, 2021.
Angela Dai, Angel X Chang, Manolis Savva, Macicj Halber, Thomas Funkhouser, and Matthias Nicßner. ScanNet: annotttnd 3D Reconstructions of Indoor Scenes. In CVPR, 2017.
David Dohan. Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A Saurous, Jascha bock-Dickstein, et al. Language Model Cascades. In ICML Workshop, 2022.
Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou. Neural Logic Machines. In ICLR, 2019.
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google Scanned Objects: A High- Quality Dataset Scanned Houschold Items. ICRA, 2022.
Mark Endo, Joy Hsu, Jiaman Li, and Jiajun Wu. Motion Question Answering via Modular Motion Programs. In ICML, 2023.
Tanmay Gupta and Aniruddha Kembhavi. Visual Programming: Compositional Visual Reasoning without Training. In CVPR, 2023.

---

Chi Han, Jiayuan Mao. Chuang Gan, Josh Tenenbaum, and Jiajun Wu. Visual Concept-Metuconeeep Learning In NeurlPS, 2019.
Yining Hong, Yilun Du, Chunru Lin, Josh Tenenbaum, and Chuang Gan. 3D Concept Grounding on Neural Fields. In NeurlPS, 2022.
Joy Hsu, Jiayuan Mao, and Jiajun Wu. NS3D: Ncuro-Symbolic Grounding of 3D Objects and Relations. In CVPR, 2023.
Shijia Huang, Yilun Chen, Jiaya Jia, and Liwei Wang. Multi Vieww Transformer for 3D Visual Grounding. In CVPR, 2022.
Drew Hudson and Christopher D Manning. Learning by Abstraction: The Neural State Machine. In NeurIPS, 2019.
Drew A Hudson and Christopher D Manning. Compositional Attention Networks for Machine Reasoning. In ICLR, 2018.
Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki. Bottom Up Top Down Detection Transformers for Language Grounding Images Point Clouds. In ECCV, 2022.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, Lawrence Zitnick, and Ross Girshick. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. In CVPR, 2017a.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Judy Hoffman, Fei-Fei, Lawrence Zitnick, and Ross Girshick. Inferring and Executing Programs for Visual Reasoning. In ICCV, 2017b.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnacve, Ishan Misra, and Nicolas Carion. MDETR Modulated Detection for End-to-End Mudai-Modal Understanding. In ICCV, 2021.
Qing Li, Siyuan Huang, Yining Hong. Yixin Chen, Ying Nian Wu, and Song-Chus Zhu. Closed Loop Neural- Symbolic Learning via Integrating Neural Perception, Grammar Parsing, and Symbolic Reasoning. In ICML, 2020.
Runtao Liu. Chenxi Liu, Yutong Bai, and Alan L. Yuille. SEEV--Ref+: Diagnosing Visual Reasoning With Referring Expressions. In CVPR, 2019.
Shilong Liu,, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, al. Grounding DINO: Marrying DINO with Grounded Pre-training for Open-set Object Detection. arXiv preprint arXiv:2303.0 054999 2023.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua Tenenhaum, and Jiajun Wu. The Neuro-Symbolic Concept
Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision. ICLR, 2019.
Jiayuan Mao, Haoyue Shi, Jiajun Wu, Roger Levy, and Joshua B. Tenenbaum. Grammar Brasdd Grounded Lexicon Learning. In NeurlPS, 2021.
Jiayuan Mao, Tomas Lozeno-Perez, Joshua Tenenbaum, and Lestie Pack Kaclbing. PDSketch: Integrated Domain Programming, Learning, and Planning. In NeurlPS, 2022.
David Mascharka, Philip Tran, Ryan Soklaski, Arjun Majumdar. Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning. In CVPR, 2018.
Charles Ruizhongtai Qi, Li Yi, Hao Su, Leonidas Guibas. PointNet+4: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. In NeurlPS, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models From Natural Language Supervision. In ICML, 2021.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. DenseCLIIP: Language-Guidedd Dense Prediction with Context-Aware Prompting. In CVPR, 2022.
Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In NewrIPS, 2015.
Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. Two-Stream Adaptive Graph Convolutional Networks for Skeleton- Beaset Action Recognition. In CVPR, 2019.
12

---

Mohit Shridhar, Lucas Manuelli, and Dieter Fox. CLIPort What and Where Pathways for Robotic Manipulation. In CoRL, 2022.
Didac Surís, Sachit Menon, and Carl Vondrick. ViperGPT: Visual Inference via Python Execution for Reasoning. In ICCV, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.1397) 2023.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal Few-Shot Learning with Frozen Language Models. NeurlPS, 2021.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: Unifying Architectures, Tasks, and Modalities Through Simple Sequence-to- Se-quunnce Learning Framework. In ICML, 2022.
Renhao Wang, Jiayuan Mao, Joy Hsu, Hang Zhao, Jiajun Wu, and Yang Gao. Programmatically Grounded, Compositionally Generalizable Robotic Manipulation. In ICLR, 2023.
Zhengyuan Yang. Songyang Zhang, Liwei Wang, and Jiebo Luo. SAT: Semantics Assisted Training Visual Grounding. In ICCV, 2021.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding. In NeurlPS, 2018.
Eric Zelikman, Qian Huang, Gabriel Poesia, Noah D Goodman, Nick Haber. Parsel: A (De)jcompositions Framework for Algorithmic Reasoning with Language Models. arXiv:2212.10561, 2023.
13

---

Supplementary for What's Left? Concept Grounding with Logic-Enhanced Foundation Models
The appendix is organized the following. In Appendix A we formally define the first-order logic programs used by LEFT as well as discuss the broader impact of our method. In Appendix [ we present experiment results with error bars and details on the construction of zero-shot transfer tasks. In Appendix C
we provide implementation, training details, and code.
A LEFT
A.1
Function definitions
We summarize the function definitions of LEFT Table 7 We note that E(expr) represents the execution result of expr. If the input to concepl functions contains concrete entity (e.g., result of iota), we take the dot product between the entity and the MLP result. the argument text (e.g.. color), the function is implemented as generative function instead of discriminative one. Ternary and high-arity relations can be generalized similarly to unary and binary relations.
Signature & Implementation
Semantics
existsfvar, expr) b. boolean
Return truc there is at least assignment of var that satisties expr.
Implementation: b = (ex(E()xp))
forullivar, expr) b: boolean
Return there all ussignments sal- isfy expr.
Implementation: b = (espp()))))
intofvar, expr) entity
Return assignment to war's that satisfies expr:
Implementation: = rofimor (exprr) (expr))
nor(espr) e: boolean
Compute the negation an expression.
Implementation: ((expr), for E 21,
and(expr expry) e: boolean
Compute conjunction expressions.
Implementation: min( (exprr E(espry))) for (1,2,".., .N)
ortexpr,, expr2) e: booleam
Compute the disjunction of two expressions.
Implementation: (.
(1, .N
vountyvar, expr) c: integer
Return count of assigmments to war's that satisties expr.
Implementation: >, (((pp)))))
greater paatter__hn(epprr... expry) - hooleam
Return score indicating whether E (expr, is greater than ((epp2))
Implementation:: a( (E(pp))) (espra)) -3)
viewfexpr)
Set the object returned by (expr) as the object agent looking at.
Implementation: E (expr)
dotexpr)
Perform action computed by E(eup).
Implementation: (expr)
vescribelvar, expr) e: text
Return the best assignment to var (e-g. colors, shapes, etc.) that describes ohject returned (eepp).
Implementation: saftmaue(& (expr))
concept(vur..) booleun Implementation: Mopoopopi (!
objects, actions, and texts as inguts and returns Boolean values.
unary function
concopt(var,... vor,)) e: boolean
objects, actions, and texts inputs and returns Boolean values.
binary function
Implementation: e. MI.pomop hines
Table 7: Function definitions for first-order logic programs in LEFT.
14

---

A.2
Broader impact
LEFT leverages pre-trained large language model as its language interpreter, and hence, even though our prompts are general examples of first-order logic, we do not have direct control over the LLM's generation. The LLM may output harmful biases, which we will highlight an important warning in our code release.
15

---

B Experiments
B.1 Error bars
We present LEFT performance with error bars, taken from three runs with different seeds. The results for CLEVR [Johnson al. 2017a]. CLEVR-I0%, and CLEVR-Human are below. All values averaged over three random seeds, and the + sign shows the computed standard error.
CLEVR-I0%
CLEVR-100%
Humanss
LEFT 0.942-n.0s
0.9964a001
0.780 / 00.694
Table 8: Results on the CLEVR dataset with standard errors. Average performance and standard errors are computed based on three random seeds.
B.2 Transfer task construction
We detail the construction processes for CLEVR-Ref, CLEVR-PuzZles, and CLEVR-RPM below.
CLEVR-Ref We generate the CLEVR-Ref dataset using subset of the templates from the CLEVR- Ref+ [Li et al. 2019] dataset. We did not directly use the CLEVR-Ref+ dataset because it contains ordinal number concepts (in particular, the first, the second, etc.) that does appear in the original CLEVR dataset. Concretely, we used the following four templates:
1. {Select, Find, Point to} the Object
2. (Select, Find, Point to} the Object (that is) Relation I the Object 2.
3. (Select, Find, Point to} the Object (that is) Relation the Object and Relation the Object 3.
4. There is a(n) Object (that is) Relation 2 the Object 3, {select, find, point to} the Object (that is) Relation I it.
Here, items such Object 1, Object 2. and Object 3 will be replaced by object-level descriptors such as "shiny cube." Relation and Relation 2 will be replaced by relational concepts the dataset.
CLEVR-Puzzles The CLEVR-Puzzles dataset is generated based on the following template: "Can you find four objects from the image such that: Object is description; Object is relation object j. Here, Object object j will be replaced by "Object 1," "Object 2," "Object 3," "Object 4;" descriptions will be replaced by object-level descriptions such as "shiny cube;" relations will be four object descriptions
replaced by relational concepts in the dataset. In our data generation, (one for each object) and three object relation descriptions. Furthermore, we make sure that none of the object-level descriptions uniquely identify object, but the global solution is unique.
CLEVR-RPM The CLEVR-RPM dataset is generated based on the following template: "There 9 objects. ordered in 3x3 grid: row col is description 1; 1 col 2 is description 2; row I col is a description 3; row 2 col is a description 4; row col 2 description 5; row col 3 is description 6; one object at row 3 col 3. Can you find an object in the scene that can there?" Here, descriptions are object-level descriptions such "shiny cube." " Following existing work on Raven's 's Progressive Matrices [Barrett et al. 2018), we randomly choose the "progression dimensions" from color, shape, material, and size for rows and columns.
col 1 description row col is a description 8; am missing
16

---

Model
C.1
Implementation details
LEFT consists of several modular components in code. The first is GPT-3.5 [Brown et al. 2020 backed language interpreter, which takes input queries from each dataset, and outputs first-order logic. The domain initializer takes the generated first-order logic programs and automatically initializes concept embeddings based on the parsed arity. The -omain--pecific grounding modules extract entity and relational features from the imput modality. Finally, the executor executes differentiable logic programs with the outputs of the grounding modules. We release our code below.
C.2 Training details
We use the official data splits released for each dataset, CLEVR [Johnson al. 2017a] ReferIt3D AAchlioptas et al. 2020]. HumanMotionQA [Endo et al. 2023 and Cliport [Shrid har al., 2022]. The core hyperparameters were set 128 for concept embedding dimensions, and learning rate taken from prior neuro-symbolic concept learning repositories that we use as baselines.
The full details are provided with the code, linked below.
C.3 Compute
We trained with NVIDIA Titan RTX per experiment for all datasets, from an internal cluster.
C.4
Code
We publicly release our code. See our project website for additional details.
17


Reviews:
[{"review_id": "40YIgGYayj", "reviewer": "Reviewer_2ce2", "summary": "The manuscript proposes a framework that leverages an LLM to map queries to executable first-order-logic programs, with domain-specific grounding functions. The manuscript includes experiments on multiple tasks and domains.", "questions": "N/A — see above", "limitations": "The manuscript does not include any sections on Limitations or Societal Impact.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is mostly well-written.\n\nThe paper considers an interesting topic, under compelling methodology (i.e., neuro-symbolism).\n\nThe presentations of visualisations and results are mostly clear.", "weaknesses": "Section 1 (Introduction) / Figure 1 — One has to be particularly careful in applications that involve some (other) embodiment, as concepts like “left” could adopt an alternative reference frame, beside just the egocentric one. How does the approach incorporate this reasoning?\n\nSection 1 (Introduction; L49-51) — Whereas no domain-specific language is required at the reasoning level, there still needs to be lexical alignment with the concepts supported by the domain-specific grounding modules. Furthermore, the approach does still need predefined built-in functions (Table 1) and  domain-specific concept names (L158).\n\nSection 3.2 (L175-177) — Let’s be exceedingly explicit here. The manuscript states that the function implementations do not require manual definition in code. How are they generated? What are all the core FOL programs? What are their mappings from concepts initialized by the LLM?\n\nTable 4 / Table 5 — The data percentages seem to be chosen arbitrarily. How were the percentages decided on? What happened for, e.g., 25% and 50%?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: Alternative reference frame for “left”.**\n\nA: We thank you for bringing up this point! In domains where we need reference frames to disambiguate concepts like “left” (e.g., in 3D), the grounding of such concepts will explicitly consider the viewpoint of the agent — usually specified in the language query. For example, in the SR3D dataset, given the query “Facing the cabinets, pick the object to its left”, Neuro-FOL will generate the following FOL formula:\n```view(Object, lambda x: cabinets(x)) and iota(Object, lambda x: object(x) and left(x, iota(Object, lambda y: cabinets(y))))```.\nThe view function is learned as a generic high-arity function, without requiring any specific implementation to change the actual viewpoint of the pointcloud. In domains where no viewpoint is specified in language, such as the 2D domain, we assume an egocentric reference frame. We will clarify this in the main text. \n\n**Q: Clarifying concepts supported by grounding modules.**\n\nA:  We clarify that Neuro-FOL does not depend on a predefined set of lexical concepts and this is an important advantage of our system compared to earlier work on neuro-symbolic learning. To achieve this, we rely on LLMs to *automatically* extract concept names from natural language instructions in the dataset; Neuro-FOL then initializes domain-specific grounding modules (in our work, MLPs) accordingly to learn the grounding of these concepts from training data. In other words, the concepts and grounding modules are *not* predefined and can support execution of any given query. We note that *domain-specific concept names* means that these concepts are grounded to particular domains (e.g., 3D point clouds) and not shared across different datasets, but they are not manually defined.\n\nNeuro-FOL does require a minimal but general set of predefined FOL functions. Important to our goal of a general neuro-symbolic framework without any predefined domain-specific functions, Neuro-FOL only requires the FOL functions in Table 1 to reason across 2D, 3D, temporal, and robotic manipulation domains. These functions are either general logic and numeric operations (such as counting, forall, exists), or functions that handle inputs and outputs (e.g., return a text description of object categories, or execute an action output by other modules).\n\nFor a new task, prior neuro-symbolic approaches require designing new domain-specific languages to support necessary operations and writing differentiable implementations for each function. In contrast, Neuro-FOL enables easy integration, while retaining all the benefits of neuro-symbolic learning. For a new task, we can simply define a neuro-symbolic system with the following code:\n\n```\ndomain = make_domain(ALL_LANGUAGE)\nparser = GeneralizedFOLPythonParser(domain)\nexecutor = GeneralizedFOLExecutor(domain, parser)\n\nscene_graph = SceneGraph{2D/3D/temporal}(THIS_INPUT_SCENE)\nwith executor.with_grounding(scene_graph):\n    parsing = parser.parse_expression(THIS_INPUT_TEXT)\n    execution = executor.execute(parsing)\n    loss = loss(execution, gt)\n```\n\nBy passing in all language input, a scene (2D, 3D, temporal, etc), and a text query, Neuro-FOL will automatically define learnable domain-specific grounding modules, generate the FOL program, and execute domain-independent functions differentiably, such that you can simply apply any downstream loss as you would an end-to-end method. We believe this is an exciting improvement over prior works that will increase accessibility and flexibility of neuro-symbolic learning. We will clarify this in the main text!\n\n**Q: Clarifying function implementations.**\n\nA: Here, what we mean is, for all concepts extracted by LLMs, we use a generic mechanism to define the corresponding grounding module. For example, if a concept is unary (e.g., ```red(x)```), the grounding module will be a MLP layer that maps object representations to a scalar in [0, 1]. For concept ```beside```, the LLM specifies that the concept takes in features of two entities, and hence Neuro-FOL creates an MLP that operates on binary features.\n\nMore specifically, given the LLM-interpreted program ```exists(Object, lambda x: dresser(x) and beside(x, iota(Object, lambda y: cabinet(y)))```, ```exist``` and ```iota``` are domain-independent FOL functions, which take as argument outputs from domain-specific functions of ```dresser```, ```cabinet```, and ```beside```.  The FOL functions have built-in implementations and are listed in full in Table 1, with implementations in Appendix A.1. The domain-specific concept grounding functions do not require manual definitions in code, and their implementations are generated using the generic mechanism described above. We will clarify in the main text.\n\n**Q: Data efficiency percentages.**\n\nA: For Table 4 and 5, we followed experiment settings of data percentages reported in prior state-of-the-art neuro-symbolic methods — NS-CL and NS3D. In Table 4, with 10% of train data, only small improvements upon end-to-end methods were seen, and hence additional percentages were not considered. For Table 5, we similarly followed prior work; due to your suggestion, we have additionally added experiment results for 25% and 50%. As expected, our work shows larger improvement upon MAC at more data efficient settings. \n\n|              | Pref-def.  |  10%  |  25%  |  50%  |  100% |\n| -----------  | ---------- | ----- | ----- | ----- | ----- |\n| NSCL         | Yes        | 0.989 | 0.992 | 0.994 | 0.992 | \n| TbD-Net      | Yes        | 0.541 | 0.560 | 0.912 | 0.991 | \n| MAC          | No         | 0.673 | 0.940 | 0.974 | 0.989 |\n| Neuro-FOL    | No         | 0.941 | 0.991 | 0.992 | 0.996 | \n\n**Q: Limitations and societal impact.**\n\nA: We include discussion in Section 4.2, and broader impacts in Appendix A.3. We agree that an extended discussion section tackling both limitations and broader impact will be helpful! We will add an additional section in the main text taking into account all comments. \n"}, {"review_id": "JrXmqGYvNf", "reviewer": "Reviewer_xPiC", "summary": "This paper proposes an approach for general-purpose language grounding across a variety of domains and tasks. The approach first uses an LLM to generate a domain-general program, which can be implemented across different domains using domain-specific functions, represented as neural networks (and domain-general implementations of certain FOL predicates). Experiments are conducted on a variety of domains, including image understanding, video understanding, and robotic manipulation.", "questions": "* Is using MLP implementations of domain-specific functions limiting in some ways? I.e., are there functions which, in experiments, are more difficult to train than others?\n* Are the implementations of the FOL primitives hardcoded (e.g. as \"min\" for \"and\")?\n* How are the domain-specific functions trained? There are few details on the training process, but I am assuming as it is end-to-end differentiable that they are trained using LLM-generated programs on some training data\n* How is the arity determined for concepts? Is it just \"proposed\" by the LLM during training?\n* I was confused about the presentation of Table 4 and 5 -- what is a data efficiency setting? Is this just the amount of training data used (presumably to train the domain-specific functions)?", "limitations": "There wasn't much discussion of limitations. I would have liked to see discussion wrt. analysis of the proposed approach, and the potential limitation of completely separating the language and perception aspects of these tasks by a single FOL program.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The approach is compelling and relatively original -- it is especially original in combining a single unified formal representation across a variety of tasks. It is obviously addressing an important problem, and having modular approaches like these is important for investigating failure cases.\n\nThe approach is quite clearly detailed, except for details in training.\n\nExperiments are very thorough. I appreciate evaluating on human data. I would have liked to see more analysis in model performance.", "weaknesses": "My main concern is that modularizing the approach into essentially two components: (a) language -> FOL, and (b) FOL + perception -> answer/action, is glossing over nuances of language use that may require interaction between the language and perception aspects. For example, the implementation of \"left\" may be dependent on the identities of the objects participating in the spatial relationship: there may be a canonical \"left\" side of an object, given its affordances (e.g., the \"left\" of a fridge may be the area to its left when one is standing facing the fridge, regardless of the actual perspective being taken in the 3D question answering task); it seems the current approach would not be able to handle this context-dependence of \"left\", unless the domain-specific implementation of \"left\" was somehow able to encode all of the information necessary to make this inference (that a particular region is a fridge, and that a fridge has a canonical \"left\"). Another place where this comes up is that the meaning of adjectives can be highly context dependent in natural language use: e.g., the meanings of color terms depend strongly on context (as a specific example: let's say there is a very pale blue object and a very pale red object -- when hearing \"blue\", one would most likely choose the pale blue object. But if the same very pale blue object is paired with a more vibrant blue object, \"blue\" is most likely going to refer to the more vibrant one). In general, I'd be interested to hear discussion about this particular problem and how (if) the proposed approach could implicitly capture these more direct relationships between language use and perception, or if FOL acts as a bottleneck which removes any ambiguity in sentence meaning (it seems this is the goal in its design -- even the program \\iota x blue(x) ^ sphere(x) in L132 may be ambiguous if there are multiple blue objects of varying shades!).\n\nI also would have liked to see more in-depth analysis of the approach. In particular, what's left for tasks where there is still a lot of room for improvement -- basically everything except CLEVR? Because the approach is modular, it seems there is a lot of room for analysis in how different parts of the approach might be failing -- are the programs incorrect? (What causes a program to be inexecutable?) Are the learned domain-specific functions erroneous?\n\nSmall nitpicks:\n* Having a consistent example in Figure 2 would be nice.\n* I'd suggest putting the FOL programs in Figure 3", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: Language and perception.**\n\nA: Thank you for bringing up these examples! There are indeed nuances that require interactions between language and vision. Similar to what you suggested, there are two candidate approaches: 1) interpreting language semantics in a context manner, by considering, e.g., the comparison classes for certain adjectives, and 2) relying on grounding modules for learning context-dependent grounding. In this paper, we choose 2) because our main focus is on integrating the unified reasoning framework and domain-specific concept learning. In general, we think option 2) has the advantage that the language interpretation part can be solely trained on text data, which is of a larger scale than vision-language data.\n\nReference frames in spatial relations are handled in our work explicitly — they are specified in instructions in our 3D dataset. For example, given “Facing the cabinets, pick the object to its left”, Neuro-FOL generates `view(λx: cabinets(x)) and iota(λx: object(x) and left(x, iota(λy: cabinets(y))))`\nand the grounding of concepts will explicitly model the viewpoint of the agent. The view function is learned as a generic high-arity function, instead of specific ways to transform the point cloud. You are correct the semantics of “left side” is object-specific. Neuro-FOL will learn a “left_side” concept that captures a canonical left side of the object from the data distribution, since the relation grounding module takes object features as input too.\n\nContexts (e.g., comparison classes) in adjective grounding is another important aspect. In the color example, Neuro-FOL can partially handle this by modeling the \"degree\" of whether each entity satisfies the concept. For example, to find *the* blue object,” the FOL execution assumes only one single object will be selected via \"softmax\", which partially handles the ambiguity. Concretely, suppose we have three objects, a pale blue, pale red, and vibrant blue object. To find *the* blue object, it would choose the “bluest” one—the vibrant blue object if compared to both others. This will also work to a certain extent, for compositional concepts such as `blue(x) ^ sphere(x)`. In other cases, since our object encodings do contain contextual information; therefore, most of the time we rely on learning from datasets to resolve such grounding.\n\nIn sum, Neuro-FOL relies on context-dependent grounding modules to handle these cases. Though we do not build in pragmatic reasoning frameworks such as RSA [1], it is a direction we are excited about. We are very interested in tasks with ambiguous language and extending Neuro-FOL to be truly probabilistic, modeling speaker intents, and leveraging feedback from perception execution. Thank you again for the great feedback!\n\n[1] Goodman, Noah D., et al. Pragmatic language interpretation as probabilistic inference. Trends in cognitive sciences (2016).\n\n**Q: Analysis of Neuro-FOL.**\n\nA: Indeed, Neuro-FOL’s modularity enables error attribution. For more difficult tasks (3D, human motion, robotics), the FOL programs are typically correct, but the domain-specific functions are more difficult to learn. For example, in ReferIt3D, there are 607 categories, with many classes visually similar (e.g., cabinet and dresser). We see that Neuro-FOL classification accuracy for concept grounding is not very high. Hence, we can potentially rely on more classification labels, instead of only language instructions, to improve performance on this task.\n\nFor syntax errors in LLM-generated programs, we can easily detect them and resample new programs. However, there is currently no way we can recover from semantic errors in the programs. We currently use GPT-3.5 in our experiments, but anticipate that GPT-4 would yield stronger performance; however, we were not able to leverage GPT-4 due to cost. Additionally, a promising future direction is to finetune the LLM from execution feedback.\n\n**Q: MLP implementations.**\n\nA: Thank you for bringing up this point! We do assume that MLPs can realize all domain-specific functions, as they take learned features as inputs, which are also jointly trained. This can occasionally be limiting if some functions are significantly more difficult to learn. In HumanMotionQA, for example, “action” concepts yield accuracy 0.637, while “body part” concepts only yield 0.495. Actions involving motion cues in multiple frames are easier to learn.\n\nWe can definitely extend such MLP implementation to use more complex NNs, which likely require more training data. Additionally, one can initialize different networks for different concepts if they have prior knowledge on the task. We can also directly integrate pretrained visual models as domain-specific grounding modules. \n\n**Q: Function implementations.**\n\nA: Yes, Neuro-FOL requires a small but general set of functions for execution, listed in Table 1. These functions are either general logic and numeric operations (e.g., counting, exists), or functions that handle inputs and outputs (e.g., return a text description, execute an action).\n\n**Q: Training process.**\n\nA: Yes, the domain-specific functions are trained through backpropagation from the downstream loss functions, as our executor is fully differentiable.\n\n**Q: Arity for concepts.**\n\nA: Yes, both concept names and arities are proposed by the LLM. For example, given `above(x, y)`, we will treat `above` as a binary relation.\n\n**Q: Data efficiency setting.**\n\nA: Yes, the percentages in Tables 4 and 5 indicate what percentage of train data compared to the full dataset. For example, for Table 4, we train on 0.5% (329 examples), 1.5% (987 examples), etc., and show improvements over end-to-end methods. \n\n**Q. Figure edits.**\n\nA: Thank you for the suggestions! We will update the paper accordingly. \n\n**Q: Limitations.**\n\nA: We agree that an extended discussion section of limitations and future work will be helpful, and will add it to the main text incorporating all comments during review.\n"}, {"review_id": "2xw3Z7K3FP", "reviewer": "Reviewer_B8BR", "summary": "The paper presents an approach for solving visual reasoning tasks across multiple domains such as 2D image QA and robotic object manipulation. The approach (Neuro-FOL) uses an LLM to generate a first-order logic (FOL) program that is executed with a combination of learned and predefined modules. One aim is to take advantage of a pre-trained LLM (e.g., GPT 3.5) to generalize to new visual reasoning tasks at inference time. Neuro-FOL is evaluated on a diverse set of benchmarks including the CLEVR 2D VQA tasks, ReferIt3D 3D spatial reasoning tasks, HumanMotionQA tasks, and the CLIPort object packing tasks. Results demonstrate that Neuro-FOL is competitive with and sometimes outperforms strong baselines.", "questions": "1. How does Neuro-FOL performance compare when training jointly on multiple domains vs. independently on individual domains?\n2. Are both (a) using FOL and (b) not having a predefined set of concepts important?\n3. How does Neuro-FOL perform on concepts that are not well represented in the training data?", "limitations": "The limitations are briefly discussed, but additional details could be provided (perhaps based on answers to the questions above). The broader impacts are discussed in the appendix.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The problem of grounding concepts across different domains (e.g., 2D images vs. 3D scenes) is an important problem because many concepts are naturally defined across these domains. This problem is well motivated in the introduction.\n- The ability of Neuro-FOL to learn new concepts that are generated by the LLM during training is neat. Practically speaking, this should save designers the time required to define these concepts.\n- The paper evaluates Neuro-FOL on a diverse set of domains including 2D image QA, 3D scene QA, motion capture QA, and robotic manipulation, which demonstrates the generality of the proposed approach. Furthermore, Neuro-FOL has strong performance in all tasks, suggesting that it will similarly do well in other domains.\n- The paper is well written and easy to follow, but some details are missing as discussed below.", "weaknesses": "- The abstract and introduction start with the grand vision of grounding the concept “left” in multiple domains. However, from the experimental evaluations, it is unclear if such “domain-independent” reasoning is (a) learned and (b) useful.\n   - (a) Are concepts learned in a domain-independent manner? In other words, is Neuro-FOL jointly trained on datasets from multiple domains? These details are not clearly stated in the current manuscript.\n   - (b) The experimental results do not clearly demonstrate if “domain-independent” grounding is useful. A specific experiment to test this would be training Neuro-FOL independently on different domains and then comparing the performance of Neuro-FOL training jointly. Such an experiment does not appear in the paper. Thus, I do not see support of these claims.\n- The proposed approach differs from LLM + API based methods (e.g., Gupta and Kembhavi, 2023) in two ways: (a) the use of first-order logic and (b) not requiring a predefined set of modules (i.e., concept networks in Neuro-FOL). It is unclear which of these two design concepts is most important. We can imagine a world in which (a) does not matter but (b) is critical. Alternatively, both (a) and (b) might be important. The experimental evaluations do not disentangle these two features. Thus, it is unclear where the important novelties in the method lies. Note: the ViperGPT experiments do not disentangle this question. Thus, it would be helpful if the authors could comment on this further.\n- While learning concept MLPs through Neuo-FOL training has the benefit that concepts do not need to be predefined, this may also be a disadvantage because there may only be a small amount of data for a given concept. For example, a given dataset may have many questions asking about “left” and “right” and may only have a few asking about “in between” or “sandwiched between”. It is unclear how the proposed approach would generalize to such situations.\n- L257: “Notably, Neuro-FOL is able perform on all domains, instead of requiring specific models for each setting.” This claim is misleading, as components of Neuro-FOL are trained for specific domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: Domain-independent reasoning & joint training.**\n\nA: We realize that there may be confusion in our descriptions about \"domain-specific\" vs. \"domain-independent\" and \"reasoning\" vs. \"grounding.\" Here, we clarify that Neuro-FOL is a framework of \"domain-independent reasoning\" (composed of an LLM interpreter and FOL executor) and \"domain-specific grounding.\" Concretely, we learn the grounding of the word \"left\" separately in different domains, because they are grounded on different modalities: 2D, 3D, temporal sequences, etc. We agree that learning a single representation for the concept \"left\" that is shared across different modalities is an interesting and important direction.\n\nTo directly answer your questions, (a) concepts are learned through a “domain-specific” grounding module based on input modality, and (b) we do not conduct “domain-independent” grounding, instead we conduct domain-independent *reasoning* through a LLM interpreter and FOL executor. All experiments are trained on different domains, but Neuro-FOL enables the same domain-independent reasoning mechanism to work on all tasks.\n\nWe do not conduct joint training across domains as (1) the domain-independent LLM interpreter and FOL executor are not trained, and (2) the domain-specific grounding modules can not be easily shared, as at the end they operate on different modalities. Because Neuro-FOL can be trained, a promising future direction is to finetune the LLM if we had access to the model weights and sufficient compute, and hence improve program generation and transfer reasoning across domains. We will clarify this in the main text!\n\nIn the global response, we have also added code samples that highlight how Neuro-FOL’s shared domain-independent components enable flexible adoption of neuro-symbolic learning for new domains and tasks. \n\n**Q: Difference from LLM + API methods & importance of (a) usage of FOL and (b) no requirement of predefined concepts.**\n\nA: Thank you for suggesting the disentanglement between the \"reasoning primitives\" and \"learnable concepts!\" The most important contribution we want to highlight in this paper is indeed (b). In many domains (e.g., human motion and robotic manipulation), it is important to have systems that can learn new concepts from available data outside of predefined functions, as mentioned in your review. Our system allows for this, and learns new concepts instead of being constrained to an inference only system. Regarding (a), it is definitely possible to extend our language with more primitives (e.g., the ones used by ViperGPT and VisProg). In this work, we chose FOL primarily because it is general, expressive, and allows us to build an end-to-end differentiable computation graph while executing the programs across all domains.\n\nPlease also see our added experiments on VisProg in addition to ViperGPT in the global response that analyzes differences between LLM + API baselines and Neuro-FOL.\n\n**Q: Sufficient data for concept learning & performance on not well represented concepts.**\n\nA: Thank you for bringing up this point! We agree that Neuro-FOL may perform poorly when there is insufficient data for training. For example, in the ReferIt3D task, the concept \"clothes\" is not well represented and appears only 18 times in the train set, and its classification accuracy in the test set is 0.207; concept \"cart\" occurs 84 times with accuracy 0.320. By contrast, more commonly seen concepts such as \"couch\" occur 3640 times with accuracy 0.920, and \"sink\" occurs 3028 times with accuracy 0.892. We note that this is not a disadvantage particular to our system, as all learning systems will face this problem when they are trained solely on the given dataset. We provide three main perspectives to this problem. \n\nFirst, Neuro-FOL is notably more data efficient than end-to-end methods due to its ability to decompose learning into modular networks and generate more training data for each individual concept. For example, given one training datapoint each for “rainbow fire hydrant” and “rainbow sink”, end-to-end methods may have one datapoint for each of the two rare concepts, while Neuro-FOL will have two datapoints for “rainbow”, and potentially more data from “fire hydrant” and “sink” learned elsewhere. This is partially reflected in the data efficiency comparisons in Table 4 and 5. \n\nSecond, to partially mitigate this issue for Neuro-FOL, we instruct the LLM interpreter to canonicalize the concepts when reasonable, hence concepts such as “in between” and “sandwiched between” may both be merged to the “between” concept by the LLM. \n\nThird, it is possible and straightfoward to incorporate externally trained models in Neuro-FOL. We can integrate pretrained models as domain-specific grounding modules, for example, directly plugging in Grounding DINO [1] for object detection. For some tasks, there are no off-the-shelf general visual recognition models (e.g., relations, 3D concepts, human motion). Therefore, we train our concept groundings by learning solely from the dataset. We definitely agree that how to better integrate other pretrained models by finetuning, or by incrementally learning new concepts, is important. Thank you for the feedback! We will clarify this in an extended discussion section in the main text. \n\n[1] Liu, Shilong, et al. \"Grounding DINO.\" 2023.\n\n**Q: Clarification of Neuro-FOL’s ability to perform across domains without requiring specific models.**\n\nA: Thank you for catching this, we will update this to without “requiring domain-specific *reasoning modules*”. Our intended goal was to highlight that Neuro-FOL’s domain-independent LLM interpreter and FOL executor are re-used on all domains, in contrast to prior works which require new models to be defined from scratch.\n\n**Q: Limitations.**\n\nA: We agree that an extended discussion section tackling limitations and future work will be helpful! We will add this in the main text."}, {"review_id": "HVTf2h2IuA", "reviewer": "Reviewer_DugZ", "summary": "This paper introduces Neuro-FOL, a method combining a large language model for generating FOL programs, an FOL executor and trainable concept grounding modules to improve performance on a number of visual/3D QA style tasks. \n\nIn the method, a LLM interpretor is used to generate a FOL program that is not specific or grounded to the exact domain. A differentiable first-order logic executor is then used to hierarchically evaluate the value of the program by using learnable, domain-specific grounding modules to ground the program in the specific domain. \n\nThe paper evaluates in 4 settings: 2D question answering, 3D referring expressions, temporal sequence reasoning and robotic manipulation. The method performs similarly to both NS and end-to-end methods in the normal data settings shown in Table 2, beats end-to-end methods in low data settings (Table 4). The paper also shows good performance on a set of held-out CLEVR tasks.", "questions": "In Table 6, is Neuro-FOL trained on the regular CLEVR task before being put in the held-out CLEVR tasks? What about [Open] Flamingo?\n\nIn general, could you more clearly state in the paper the training loop for Neuro-FOL. I sort of infer that it is trained end-to-end on the target tasks, but that the only parts of the network that are actually updated are the domain specific grounding modules? Is that correct?\n\nHow does your method respond to very rate concepts, if you need to finetune an MLP for every grounding function. Do you see cases where a concept has very few examples during training and that module performs poorly?\n\nHow often does the LLM fail to provide a valid program and how often does it fail to provide a correct program? Does an invalid program result in Neuro-FOL failing to run?\n\nIf the program is incorrect during training, does this hurt performance in eval?\n\nHow robust is the method in general to program failures?\n\nIn general, what are the failure cases of this method?", "limitations": "I don't think that the paper does much to address the limitations of their methodology (see above) which I think would add a lot of value to the paper.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "I think main idea of the paper is really clever and well motivated. I think using a pre-trained LLM to do the logical breakdown of the query by generating a program, which can then be differentially trained is really smart. It gives you a way to have both the generality you get from LLMs having seen many concepts and the flexibility of adapting to a new domain.\n\nThe wide variety of domains and datasets evaluated is really good. Shows that the method works across a wide variety of different domains.\n\nShows really good performance compared to end-to-end methods in the rare data case, which is good.\n\nPaper is pretty clearly presented and motivated.", "weaknesses": "I think the claim of it being a universal language for all domains and therefore generally applicable across domains is maybe misleading and not totally supported. I think I agree that you could generate a program for pretty much any query-based problem, but it sort of missing a critical assumption, that the method has sufficient data to train each of the domain-specific module that you would generate. And of course, if during evaluation, if the program generated a module that it hadn't seen during training, it would fail. I think this is a real limitation of the method that should be more clearly mentioned in the paper.\n\nRelated to the above point, the set of domains is somewhat limited, and especially the kinds of reasoning here are pretty narrow. For instance, the 2D question answering is all done on CLEVR where the visually domain is extremely limited (only a set number of different block objects in a set number of colors) and a rather limited space of kinds of questions (questions chaining together reasoning about objects color, relative position, count etc). There are for instance many kinds of questions about images (for instance in the VQA dataset) that might not really have good coverage in the training set to work. If you had to train a new MLP classifier for each kind of attribute, object class and relation you might find in VQA, it's not totally obvious to me that this method would have enough data to actually work.  SR3D is definitely more visually diverse in the number of object types, but the question types are also quite limited to these kinds of spatial reasoning questions. Similarly, robotic manipulation is more visually diverse and in the number of objects, but also suffers from a lack of diversity in the kinds of queries it needs to perform.\n\nI get that the advantage of this method over pure NS approaches is that you do not need to manually define a domain-specific language, but the comparisons don't really any improvement over them. The claims about universality would be stronger if the domains were less constrained, showing for instance that this method can deal with unusual domain groundings in a way that would be hard to anticipate for a designer for NS methods.\n\nKind of a minor point, but I think the point on line 105 about VLMs not working on different domains such as 3D scenes and motion sequences is pedantic. Models not trained on 3D scenes and motion sequences of course don't generalize to these, but one could train a network on these domains as well. For instance: Gato (S Reed 2022) is trained on multiple kinds of modalities.\n\n\"Flamingo\" in Table 6 should REALLY be stated to be Open-Flamingo. This is especially confusing since the paper uses the citation when it names the method. But it's not using the Flamingo model, it's using an open-source alternative to the model not Flamingo itself, which is very confusing/misleading.\n\nA couple of unanswered questions I had (see below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: Sufficient data for training grounding modules.**\n\nA: Thank you for bringing up this point! We agree that Neuro-FOL may fail with insufficient data, which is a limitation for most learning systems trained solely on one given dataset. We provide three main perspectives here. \n1. Neuro-FOL is notably more data efficient than end-to-end methods due to its modular structure (decomposing complex concepts into primitive ones).\n2. To partially mitigate this issue, Neuro-FOL instructs LLMs to canonicalize concepts when reasonable (e.g., \"in between\" and \"sandwiched\" may be interpreted to \"between\"), which helps learning certain rare concepts.\n3. It is possible and straightfoward to incorporate externally trained modules. For example, we can plug in Grounding DINO [1] for object detection. However, for some tasks, there are no off-the-shelf general visual recognition models (e.g., relations and human motions); therefore, we train grounding modules solely from the dataset. We fully agree that how to better integrate pretrained models by finetuning, or by incrementally learning new concepts, is important. And indeed as you mentioned, if Neuro-FOL sees a completely new concept during evaluation, it will fail, similar to other methods trained solely on the given data. We will clarify this limitation in the main text!\n\n[1] Liu, Shilong, et al. \"Grounding DINO.\" 2023.\n\n**Q: Rare concepts.**\n\nA: As mentioned above, Neuro-FOL’s modularity enables data-efficient learning of rare concepts: given one datapoint each for “rainbow fire hydrant” and “rainbow sink”, end-to-end methods may have one datapoint each of the two concepts, while Neuro-FOL will have two datapoints for “rainbow”, and potentially leveraging data for “fire hydrant” and “sink” elsewhere. This is partially reflected in our data efficiency comparisons. Yet without enough data, Neuro-FOL’s performance will indeed suffer. For example, in ReferIt3D, the concept \"clothes\" appears only 18 times in training, and its classification accuracy is 0.207. Similarly: \"cart\", 84 times, accuracy 0.320. By contrast, \"couch\", 3640 times, 0.920, and \"sink\", 3028 times, accuracy 0.892.\n\n**Q: Diversity of evaluation domains.**\n\nA: We agree that the long-tail problem is important, and difficult. However our goal here is to integrate domain-specific concept learning with a unified, domain-independent reasoning framework, hence we defer the integration of few-shot learning methods to future work. We also note that some domains we evaluate on do have complex reasoning structures; in NR3D (Table 3) we see both visual and question diversity. Complex query examples include: “It's the middle towel out of the 3 that's all beside each other, disregard the only towel”, “If you are sitting in bed with your head against the headboard, it is the nightstand to your left.”\n \n**Q: Universality of FOL.**\n\nA: Thank you for raising this point. We do not claim to outperform prior neuro-symbolic methods as they represent \"oracle\" implementations for particular datasets. We instead focus on a unified framework across domains and tasks — with the benefits of neuro-symbolism, and minimizes the need for domain-specific knowledge.\n\nIn the global response, we have also added code samples that highlight how Neuro-FOL’s unified reasoning modules enable flexible neuro-symbolic learning across domains.\n\nAdditionally, in the CLEVR-Puzzle task (Section 4.2), we demonstrate Neuro-FOL in a setting where the language instruction cannot be parsed by the original CLEVR DSL, however, it can be represented by combining concepts with our general FOL, showing that Neuro-FOL can handle new tasks that are hard to anticipate for original DSL designers.\n\n**Q: Training data for held-out CLEVR tasks.**\n\nA: Yes, Neuro-FOL is trained on the original CLEVR with 70,000 images; Open Flamingo is trained on LAION 2B with 5 million image-text examples from Multimodal C4. \n\n**Q: Training loop.**\n\nA: Yes, the domain-specific functions (feature extractors and concept MLPs) are trained through backpropagation, as our executor is fully differentiable, and we use LLMs inference-only.\n\n**Q: Program generation.**\n\nA: LLMs occasionally fail to provide valid programs for complex natural language queries: (rate 0.102 for CLEVR-Humans, 0.088 for NR3D). An invalid program will cause Neuro-FOL to fail. In the global response, we further compare this with VisProg and ViperGPT (both use LLMs for program generation). Neuro-FOL produced syntactically valid programs for 100% of queries in CLEVR-RPM (c.f. ViperGPT's 40%). For semantic correctness, since we can not automatically label programs, we evaluate execution results.\n\n**Q: Effect of program correctness & program robustness.**\n\nA: If the program is incorrect, it is either (1) not executable, hence we lose a datapoint, or (2) semantically incorrect, which will harm performance. In our experience, Neuro-FOL programs have high semantic accuracy across tasks. For syntax errors, we can easily detect and resample. However, there is currently no way to recover from semantic errors. We believe a promising direction is to finetune the LLM from execution feedback.\n\n**Q: Failure cases.**\n\nA: In our experiments, FOL programs are typically correct, but the domain-specific functions are more difficult to learn. For example, in ReferIt3D, there are 607 categories, with many classes visually similar (e.g., cabinet and dresser). We see that Neuro-FOL classification accuracy for concept grounding is not very high. Hence, we can potentially rely on more classification labels, instead of only language instructions, to improve performance on this task. Additionally as mentioned, program failures and rare concepts also decrease performance.\n\n**Q: VLMs across domains & Open-Flamingo.**\n\nA: Thank you for the suggestions, we will clarify!\n\n**Q: Limitations.**\n\nA: We agree that an extended discussion section tackling limitations and future work will be helpful! We will add this to the main text.\n"}]

Author rebuttal:
We thank all the reviewers for their constructive feedback! We have added additional data efficiency experiments on CLEVR, baseline results from VisProg & ViperGPT for comparison to Neuro-FOL, statistics and analyses of Neuro-FOL performance from the HumanMotionQA and ReferIt3D tasks, and code examples to exemplify flexibility of Neuro-FOL usage.

Many reviewers suggested extending our discussion section in the paper, with which we fully agree. We will add a lengthened discussion section in the main text, with focus on limitations, future work, and broader impact, integrating all comments from your reviews. 

Additionally, we’d like to highlight Neuro-FOL’s contribution over prior neuro-symbolic approaches. The existing methods require predefined “oracle” domain-specific functions for a given domain and task. For a new task, this means designing new domain-specific languages to support necessary operations and writing differentiable implementations for each function. This requires both domain-specific knowledge as well as neuro-symbolic background. In contrast, Neuro-FOL enables easy integration without either of the above requirements, while retaining all the benefits of neuro-symbolic learning. For a new task, we can simply define a neuro-symbolic system with the following code:

```
domain = make_domain(ALL_LANGUAGE)
parser = GeneralizedFOLPythonParser(domain)
executor = GeneralizedFOLExecutor(domain, parser)

scene_graph = SceneGraph{2D/3D/temporal}(THIS_INPUT_SCENE)
with executor.with_grounding(scene_graph):
    parsing = parser.parse_expression(THIS_INPUT_TEXT)
    execution = executor.execute(parsing)
    loss = loss(execution, gt)
```

By passing in all language input, a scene (2D, 3D, temporal, etc), and a text query, Neuro-FOL will automatically define learnable domain-specific grounding modules, generate the FOL program, and execute domain-independent functions differentiably, such that you can simply apply any downstream loss as you would an end-to-end method. We believe this is an exciting improvement over prior works that will increase accessibility and flexibility of neuro-symbolic learning.

Based on reviewer suggestions, we also ran experiments on VisProg in addition to the ViperGPT, in order to better analyze differences between LLM + API baselines and Neuro-FOL. Our method significantly outperforms baselines that leverage programs from LLMs, as Neuro-FOL has less predefined functions and more learned programs.

|                       | CLEVR-Ref  | CLEVR-Puzzles  |   CLEVR-RPM   |
| -----------       | ------ | ------ | -----|
| Neuro-FOL    | **0.94**   | **0.75**   | **0.87** | 
| Flamingo 4-shot    | N/A   | 0.54   | 0.52 | 
| Flamingo 8-shot    | N/A   | 0.57   | 0.52 | 
| VisProg    | N/A   | 0.27   | 0.51 | 
| ViperGPT    | N/A   | 0.34   | 0.04 | 

We highlight some examples where Neuro-FOL improves upon these methods. For example, ViperGPT does not contain predefined programs for “left” and “right”, hence, when running inference, the LLM produces Python code to compare the pixel coordinates in order to find such relations, which does not take into account depth of the scene, sizes of the objects, etc. By contrast, Neuro-FOL is able to learn programs for “left” and “right” respectively for more faithful execution. As an additional example, VisProg often passes in full phrases to its predefined methods, such as ```LOC(image=IMAGE, object='small yellow metal cylinder')```, instead of composing modular concepts for "small", "yellow", "metal", and "cylinder" as Neuro-FOL does, which allows for better error attribution. In addition, for more complex domains such as 3D and temporal human motion sequences, it is unclear what predefined models should be used for LLM + API methods. 

We thank you for all the helpful comments, and am happy to answer any follow-up questions! 


Meta-review: The paper is concerns with the use of LLMs for answering questions about data across different domains. It leverages LLMs to convert the questions to a logic-based domain-independent representation that is subsequently grounded in a domain of interest.

The paper has received 1 x Accept, 2 x Weak Accept, and 1 x Borderline Accept.

All reviewers appreciate the approach. The all hail the idea of using a domain independent formalism (“compelling methodology” (2ce2), “compelling and relatively original” (xPiC), “really clever and well motivated” (DugZ)). In particular, the reviewers find of value the ability to have general and formal representation of queries and questions, powered by LLMs. And then the ability to apply it to any domain afterwards with domain specific grounding. Further, the reviewers find the authors address an important and timely problem. The evaluation is compelling and convincing. Hence, the AC in consultation with the SAC accepts the paper to Neurips.

We will encourage the authors to incorporate some of the feedback given in the rebuttals. In particular, clarification re domain specific and domain independent, challenges and benefits in lack of enough domain data, etc.
Decision: Accept (poster)