SlowGuess/NeurIPS_2023_OCR huggingface dataset:
methods on concept learning across four domains. Our model excels at task performance and also data efficiency, without requiring any domain-specific knowledge. Next, in Section 4.2, we present LEFT's zero-shot generalization performance to three novel and challenging visual reasoning tasks, and compare its reasoning capabilities to that of LLM-based decomposition frameworks and a general vision-language model. Implementation and training details for LEFT can be found in the Appendix.
4.1 Concept learning across domains
We train and validate LEFT on six different datasets, grouped into four domains (Figure 3).
2D visual concept learning. In the first domain, we focus on the task of visual question answering in 2D images: given an image and a language query (e.g., "How many small cylinders are partially hidden?"), the goal is to predict an answer from a set vocabulary. The accuracy is calculated as the exact match to ground truth answers. We validate LEFT on the CLEVR and CLEVR-Humans dataset [Johnson et al., 2017a], which contain images with objects rendered of different shape, size, material, and color. CLEVR contains synthetically generated queries, while the CLEVR-Humans is a human-annotated dataset with diverse natural language questions.
3D visual concept learning. The second domain evaluates 3D referring expression comprehension. The input is a set of object point clouds in the scene, and a query (e.g. "The monitor that you would class as in the middle of the other two"). Models are evaluated by the accuracy of selecting the correct target object. In the 3D domain where we need reference frames to disambiguate relations like left, the grounding of such concepts will consider the viewpoint specified in the language query, through a generic high-arity function that does not require a specific implementation. We evaluate LEFT on the ReferIt3D dataset [Achlioptas et al., 2020], with 3D point clouds from ScanNet [Dai et al., 2017], a collection of indoor scenes. ReferIt3D consists of two subsets, SR3D, which focuses on spatial reference with synthetically generated queries, and NR3D, with human-annotated queries. We use the NR3D subset from NS3D, which contain the same set of concepts as in SR3D.
Motion capture concept learning. The third domain contains human motion capture. In particular, we focus on the task of question answering in temporal motion sequences. Given a sequence of human motion, parameterized by joints, and a question (e.g. "What action does the person do before they move backward and after they kick?"), the goal is to predict the exact match answer. We train and evaluate LEFT on the HumanMotionQA dataset [Endo et al., 2023], which asks questions about temporal relations and a variety of attributes, including action, direction, and body parts.
Robotic manipulation. Finally, we demonstrate that LEFT can learn action concepts that correspond to robotic motions, in the context of robotic manipulation. The input to this task is an image representing a tabletop environment, and a language instruction (e.g., "Pack the yoshi figure into the brown box.") The goal is to generate a sequence of actions that maximizes the average total reward over this robotic manipulation task. We validate LEFT on the CLIPort dataset [Shridhar et al., 2022], focusing on the task of sequential packing Google Scanned Objects [Downs et al., 2022].
4.1.1 Accuracy
We compare LEFT against two categories of top-performing models across all six datasets. The first category contains neuro-symbolic models, which require a pre-defined domain-specific language and implementations for each program. The second category contains monolithic end-to-end methods which usually require a significant amount of data to train. Notably, our unified LEFT framework is able to perform on all domains, instead of requiring specific reasoning modules for each setting.
Pre-def.	CLEVR	SR3D	HumanMotionQA	Cliport
Prior NS	Yes	0.992 (NSCL)	0.627 (NS3D)	0.578 (NSPose)	0.797 (ProgramPort)
End-to-End	No	0.989 (MAC)	0.670 (BUTD†)	0.430 (MotionCLIP)	0.708 (Cliport)
LEFT	No	0.996	0.620	0.582	0.793
Table 2 summarizes the results of different models on four datasets that involve synthetically generated language, and hence ground truth programs exist for neuro-symbolic methods and our approach. We see that across CLEVR, SR3D, HumanMotionQA, and CLIPort, LEFT achieves comparable performance to prior neuro-symbolic works [Mao et al., 2019, Hsu et al., 2023, Endo et al., 2023, Wang et al., 2023], and outperforms end-to-end methods in CLEVR, HumanMotionQA, and CLIPort [Hudson and Manning, 2018, Jain et al., 2022, Shridhar et al., 2022]. Importantly, our LEFT model applies a unified language interpretation and program execution framework for across all domains and tasks, while prior work on neuro-symbolic learning requires domain-specific languages and reasoning modules for each dataset. This suggests that first-order logic language can serve as an effective and general language for representing language queries.
In Table 3, we examine LEFT on tasks with human-annotated natural language queries. The first number for LEFT represents accuracy out of all queries where the program generated by LLM is executable, while the latter number represents accuracy out of all queries, considering the percentage of LLM outputs that are not executable.
Table 2: Results comparing LEFT to prior top-performing neuro-symbolic and end-to-end works. Our unified LEFT framework yields strong performance on all settings without any pre-defined programs.
Pre-def.	CLEVR-Humans	NR3D
Prior NS	Yes	0.678 (NSVQA)	0.526 (NS3D)
End-to-End	No	0.817 (MDETR)	0.430 (MVT)
LEFT	No	0.788 / 0.698	0.496 / 0.485
We note that MDETR [Kamath et al., 2021] outperforms LEFT on CLEVR-Humans, possibly because it can better capture the similarity between similar concepts such as front and in-front-of. In contrast, the representations for different concepts in LEFT are independent. The prior neuro-symbolic method we compared against outperforms LEFT on NR3D, as it also leverages an LLM for language interpretation, but uses a set of domain-specific examples in prompts that are hand-crafted.
4.1.2 Data efficiency
We evaluate LEFT on data efficiency settings for CLEVR [Johnson et al., 2017a] and SR3D [Achlioptas et al., 2020]. LEFT significantly outperforms all end-to-end methods at every data efficient percentage point tested, and performs comparably to best-performing neuro-symbolic methods [Hsu et al., 2023, Jain et al., 2022, Huang et al., 2022, Yang et al., 2021, Mao et al., 2019, Mascharka et al., 2018, Hudson and Manning, 2018] (See Table 4 and Table 5). The percentages indicate the percentage of train data used for training compared to the full dataset.
Table 3: Comparisons on natural language tasks.
Pre-def.	0.5%	1.5%	2.5%	5%	10%	100%
NS3D	Yes	0.426	0.520	0.556	0.591	0.600	0.627
BUTD-DETR	No	0.083	0.158	0.259	0.395	0.528	0.670
MVT	No	0.161	0.275	0.322	0.380	0.491	0.645
SAT	No	0.172	0.260	0.298	0.330	0.362	0.579
LEFT	No	0.410	0.521	0.565	0.579	0.591	0.620
Table 4: Results on SR3D in data efficient settings. LEFT performs comparably to prior neurosymbolic works, and outperforms end-to-end works significantly.
Focusing on CLEVR as an ablation study, we note that NSCL outperforms LEFT on the  $10\%$  setting. We attribute this gap to the additional domain-specific inductive biases that NSCL leverages. In particular, NSCL assumes the existence of four attribute spaces (color, shape, material, and size), and each concept lies in exactly one attribute space. Furthermore, all concepts within the same attribute
$^{\dagger}$  BUTD operates on a constrained setting, assuming 485 classes instead of the full 607 classes in ReferIt3D.
space are mutually exclusive. Therefore, the inductive biases encoded by NSCL can be seen as a combination of a hierarchical program-based reasoning mechanism, and additional domain-specific priors on concepts. By contrast, LEFT only leverages the program-based reasoning mechanism, which is general across many domains and tasks. In comparison of our model with other baselines and NSCL, we show that this general program-based reasoning mechanism is powerful enough to explain most of the data efficiency improvements.
4.2 Reasoning generalization across tasks
Finally, we present LEFT's ability to zero-shot generalize to unseen tasks.
CLEVR reasoning tasks. We generate three zero-shot transfer tasks based on the CLEVR datasets [Johnson et al., 2017a], with 100 evaluation examples each. See Figure 4 for an example of each task. The CLEVR
Pre-def.	10%	25%	50%	100%
NSCL	Yes	0.989	0.992	0.994	0.992
TbD-Net	Yes	0.541	0.560	0.912	0.991
MAC	No	0.673	0.940	0.974	0.989
LEFT	No	0.941	0.991	0.992	0.996
Table 5: Results on CLEVR in data efficient settings.
Ref task consists of a CLEVR image and language queries, requiring LEFT to locate the exact target object. The CLEVR-Puzzle task consists of images paired with visual puzzle descriptions, which necessitates a variable assignment puzzle solving. The CLEVR-RPM task is based on Raven's Progressive Matrices, which consists of an image and a 3x3 grid describe by language. This task requires the LLM interpreter to reason about patterns in the grid by row and column to determine the attributes of the target object, and generate queries to the FOL executor to check if such an object exists in the scene. We include dataset details in the Appendix.
Figure 4: The three zero-shot generalization tasks. As seen in the CLEVR-Ref example, LEFT's domain-independent modules resolve the language into first-order logic and generate the answer.
Query: There are 9 objects, ordered in a  $3 \times 3$  grid: row 1 col 1 is a green cube; row 1 col 2 is a green sphere; row 1 col 3 is a green cylinder; row 2 col 1 is a brown cube; row 2 col 2 is a brown sphere; row 2 col 3 is a brown cylinder; row 3 col 1 is a cyan cube; row 3 col 2 is a cyan sphere; I am missing one object at row 3 col 3. Can you find an object in the scene that can fit there?
We train LEFT on the CLEVR question answering task, and report zero-shot transfer performance on the three unseen reasoning tasks. In Table 6, we see that LEFT achieves strong performance on all datasets. We compare LEFT performance with ground truth programs (GT programs) and with FOL programs from our LLM interpreter (LLM programs). This ablation demonstrates that there are only small drops in performance with LLM programs as compared to ground truth programs across all three complex reasoning tasks. LEFT can zero-shot transfer to novel visual reasoning tasks with LLM generated first-order logic, and effectively reuse concept embeddings, enabling flexible
CLEVR-Ref	CLEVR-Puzzles	CLEVR-RPM
LEFT + GT programs	1.00	0.92	1.00
LEFT + LLM programs	0.94	0.75	0.87
VisProg [Gupta and Kembhavi, 2023]	0.35	0.27	0.51
ViperGPT [Surís et al., 2023]	0.08	0.34	0.04
OpenFlamingo 4-shot [Awadalla et al., 2023]	N/A	0.54	0.52
OpenFlamingo 8-shot	N/A	0.57	0.52
Table 6: Zero-shot generalization of LEFT to three unseen tasks; LEFT recomposes domain-independent programs and learned concepts with the LLM interpreter for flexible transfer.
generalization. Importantly, we see that LEFT performs well on CLEVR-RPM, a challenging task which contains complex Raven's Progressive Matrices [Barrett et al., 2018] descriptions that require the LLM interpreter to reason about patterns described by language. While the prompts given to the LLM are simple, our interpreter can solve complex tasks.
As baselines, we compare LEFT with LLM-based decomposition frameworks and general vision-language models. VisProg [Gupta and Kembhavi, 2023] and ViperGPT [Surís et al., 2023] are inference only, integrated LLM with APIs frameworks, that has shown success on a variety of visual reasoning tasks. In Table 6, we see that LEFT significantly outperforms VisProg and ViperGPT on the CLEVR transfer tasks. The LLM-based decomposition frameworks often fail to output executable code for complex queries.; in addition, the accuracy of execution by the APIs is low. For example, on the CLEVR-RPM task, only 0.40 of queries were executed successfully by ViperGPT, with accuracy of 0.10 out of executable queries. As these methods execute programs with a constrained set of pre-defined functions, the accuracy for executing undefined relations, such as to the right of, which is learned in LEFT, is low. In addition, for complex domains such as 3D and temporal human motion sequences, it is unclear what pre-defined models should be used for these inference-only frameworks.
For comparison to general vision-language models, we evaluate OpenFlamingo [Awadalla et al., 2023], an open-sourced version of Flamingo [Alayrac et al., 2022]. OpenFlamingo is a VL model backed by a CLIP vision encoder [Radford et al., 2021] and a LLAMA language encoder [Touvron et al., 2023]. On both 4- and 8-shot variants, LEFT outperforms OpenFlamingo significantly. Notably, OpenFlamingo cannot predict answers for CLEVR-Ref, which requires an object index as output.
We have presented the Logic-Enhanced Foundation Model (LEFT), a unified concept learning and reasoning framework that flexibly learns concepts across different domains, and generalizes reasoning to unseen, complex tasks. Our framework combines foundation models and first-order logic executors as domain-independent reasoning modules, and learnable grounding modules for specific domains. LEFT represents a step towards modeling general reasoning capability in a rightfully grounded way.
LEFT leverages pre-trained LLMs for domain-independent reasoning. Hence, the results of our system are limited by the quality of the LLM, which we cannot finetune. For syntax errors in LLM-generated programs, we can easily detect them and resample. However, we currently cannot recover from semantic errors in the programs. A future direction would be to incorporate execution traces or failures as additional inputs to LLMs such that they can fix the queries they generate.
LEFT may also fail with insufficient domain data, if it cannot effectively train the domain-specific grounding modules. We note that this is a limitation for most learning systems trained solely on one given dataset. To partially mitigate this issue, LEFT instructs LLMs to canonicalize concepts when reasonable, which helps learning certain rare concepts. It is also straightforward to incorporate externally trained modules, by directly integrating pretrained visual models as domain-specific grounding modules. For example, we can plug in Grounding DINO for object detection if we have prior task knowledge [Liu et al., 2023]. However, for some domains, there are no off-the-shelf visual recognition models (e.g., relations and human motions); therefore, LEFT can train its concept groundings by learning from data. Notably, LEFT is also more data efficient compared to end-to-end methods due to its modular structure, as it can decompose complex concepts into primitive ones.
Our framework currently does not model the interaction between the language and perception. Instead, LEFT relies on context-dependent grounding modules which learn from data to resolve ambiguous language. Future directions for LEFT include building in pragmatic reasoning frameworks, modelling speaker intents, and leveraging feedback from perception execution.
Acknowledgments. We thank Weiyu Liu and Gabriel Poesia for providing valuable feedback on the paper. This work is in part supported by the Stanford Institute for Human-Centered Artificial Intelligence (HAI), ONR MURI N00014-22-1-2740, ONR N00014-23-1-2355, Air Force Office of Scientific Research (AFOSR) YIP FA9550-23-1-0127, the Center for Brain, Minds, and Machines (CBMM), the MIT Quest for Intelligence, MIT-IBM Watson AI Lab, Analog Devices, Autodesk, J.P. Morgan, and Salesforce. JH is supported by the Knight Hennessy Scholarship and the NSF Graduate Research Fellowship.
Panos Achlioptas, Ahmed Abdelreehm, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. ReferIt3D: Neural Listeners for Fine-Grained 3D Object Identification in Real-World Scenes. In ECCV, 2020.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a Visual Language Model for Few-Shot Learning. In NeurIPS, 2022.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Learning toCompose Neural Networks for Question Answering. In NAACL-HLT, 2016.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. OpenFlamingo: An Open-source Framework for Training Large Autoregressive Vision-language Models. arXiv preprint arXiv:2308.01390, 2023.
David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Measuring Abstract Reasoning in Neural Networks. In ICML, 2018.
Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. Prompting Is Programming: A Query Language for Large Language Models. In PLDI, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language Models are Few-Shot Learners. In NeurIPS, 2020.
Wenhu Chen, Zhe Gan, Linjie Li, Yu Cheng, William Wang, and Jingjing Liu. Meta Module Network for Compositional Visual Reasoning. In WACV, 2021a.
Zhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan-Yee Kenneth Wong, Joshua B Tenenbaum, and Chuang Gan. Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning. In ICLR, 2021b.
Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzhu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, et al. Binding Language Models in Symbolic Languages. In ICLR, 2023.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying Vision-and-Language Tasks via Text Generation. In ICML, 2021.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. In CVPR, 2017.
David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A Saurous, Jascha Sohl-Dickstein, et al. Language Model Cascades. In ICML Workshop, 2022.
Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou. Neural Logic Machines. In ICLR, 2019.
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items. In ICRA, 2022.
Mark Endo, Joy Hsu, Jiaman Li, and Jiajun Wu. Motion Question Answering via Modular Motion Programs. In ICML, 2023.
Tanmay Gupta and Aniruddha Kembhavi. Visual Programming: Compositional Visual Reasoning without Training. In CVPR, 2023.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. CLIPort: What and Where Pathways for Robotic Manipulation. In CoRL, 2022.
Dídac Surís, Sachit Menon, and Carl Vondrick. ViperGPT: Visual Inference via Python Execution for Reasoning. In ICCV, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971, 2023.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal Few-Shot Learning with Frozen Language Models. In NeurIPS, 2021.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework. In ICML, 2022.
Renhao Wang, Jiayuan Mao, Joy Hsu, Hang Zhao, Jiajun Wu, and Yang Gao. Programmatically Grounded, Compositionally Generalizable Robotic Manipulation. In ICLR, 2023.
Zhengyuan Yang, Songyang Zhang, Liwei Wang, and Jiebo Luo. SAT: 2D Semantics Assisted Training for 3D Visual Grounding. In ICCV, 2021.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding. In NeurIPS, 2018.
Eric Zelikman, Qian Huang, Gabriel Poesia, Noah D Goodman, and Nick Haber. Parsel: A (De-)compositional Framework for Algorithmic Reasoning with Language Models. arXiv:2212.10561, 2023.
Supplementary for What's Left? Concept Grounding with Logic-Enhanced Foundation Models
The appendix is organized as the following. In Appendix A, we formally define the first-order logic programs used by LEFT as well as discuss the broader impact of our method. In Appendix B, we present experiment results with error bars and details on the construction of zero-shot transfer tasks. In Appendix C, we provide implementation, training details, and code.
A LEFT
A.1 Function definitions
We summarize the function definitions of LEFT in Table 7. We note that  $\mathcal{E}(\text{expr})$  represents the execution result of expr. If the input to concept functions contains a concrete entity (e.g., result of iota), we take the dot product between the entity and the MLP result. If the argument is text (e.g., color), the function is implemented as a generative function instead of a discriminative one. Ternary and high-arity relations can be generalized similarly to unary and binary relations.
Signature & Implementation	Semantics
exists(var, expr) → b: boolean	Return true if there is at least one assignment of var that satisfies expr.
Implementation: b = max(E expr))
forall(var, expr) → b: boolean	Return true if there is all assignments to var satisfy expr.
Implementation: b = min(E expr))
iota(var, expr) → e: entity	Return an assignment to var's that satisfies expr.
Implementation: e = softmax(E expr))
not expr) → e: boolean	Compute the negation of an expression.
Implementation: ei = -E expr)i, for all i ∈ {1, 2, ···, N}
and expr1, expr2) → e: boolean	Compute the conjunction of two expressions.
Implementation: ei = min(E expr1)i, E expr2i), for all i ∈ {1, 2, ···, N}
or expr1, expr2) → e: boolean	Compute the disjunction of two expressions.
Implementation: ei = max(E expr1)i, E expr2i), for all i ∈ {1, 2, ···, N}
count(var, expr) → c: integer	Return the count of assignments to var's that satisfies expr.
Implementation: c = ∑i [σ(E expr)i]
greater Than expr1, expr2) → b: boolean	Return a score indicating whether E expr1) is greater than E expr2).
Implementation: b = σ(τ · (E expr1) - E expr2) - γ)
view expr)	Set the object returned by E expr as the object that the agent is looking at.
Implementation: E expr)
do expr)	Perform action computed by E expr.
Implementation: E expr)
describe(var, expr) → e: text	Return the best assignment to var (e.g., colors, shapes, etc.) that describes the object returned by E expr.
Implementation: e = softmax(E expr))
concept(varx) → e: boolean	A unary function that takes objects, actions, and texts as inputs and returns Boolean values.
Implementation: ei = MLPconcept(fi unary)
concept(varx, vary) → e: boolean	A binary function that takes objects, actions, and texts as inputs and returns Boolean values.
Implementation: ei,j = MLPconcept(fj,i,j)
Table 7: Function definitions for first-order logic programs in LEFT.
LEFT leverages a pre-trained large language model as its language interpreter, and hence, even though our prompts are general examples of first-order logic, we do not have direct control over the LLM's generation. The LLM may output harmful biases, which we will highlight as an important warning in our code release.
B.1 Error bars
We present LEFT performance with error bars, taken from three runs with different seeds. The results for CLEVR [Johnson et al., 2017a], CLEVR-  $10\%$  , and CLEVR-Humans are below. All values are averaged over three random seeds, and the  $\pm$  sign shows the computed standard error.
CLEVR-10%	CLEVR-100%	CLEVR-Humans
LEFT	0.942±0.008	0.996±0.001	0.780±0.003 / 0.694±0.001
Table 8: Results on the CLEVR dataset with standard errors. Average performance and standard errors are computed based on three random seeds.
B.2 Transfer task construction
We detail the construction processes for CLEVR-Ref, CLEVR-Puzzles, and CLEVR-RPM below.