SlowGuess/NeurIPS_2023_OCR huggingface dataset:
CLEVR-Ref We generate the CLEVR-Ref dataset using a subset of the templates from the CLEVR-Ref+ [Liu et al., 2019] dataset. We did not directly use the CLEVR-Ref+ dataset because it contains ordinal number concepts (in particular, the first, the second, etc.) that does appear in the original CLEVR dataset. Concretely, we used the following four templates:
1. {Select, Find, Point to} the Object 1.
2. {Select, Find, Point to} the Object 1 (that is) Relation 1 the Object 2.
3. {Select, Find, Point to} the Object 1 (that is) Relation 1 the Object 2 and Relation 2 the Object 3.
4. There is a(n) Object 2 (that is) Relation 2 the Object 3, {select, find, point to} the Object 1 (that is) Relation 1 it.
Here, items such as Object 1, Object 2, and Object 3 will be replaced by object-level descriptors such as "shiny cube." Relation 1 and Relation 2 will be replaced by relational concepts in the dataset.
CLEVR-Puzzles The CLEVR-Puzzles dataset is generated based on the following template: "Can you find four objects from the image such that: Object  $i$  is description; Object  $i$  is relation object  $j$ ." Here, Object  $i$ , object  $j$  will be replaced by "Object 1," "Object 2," "Object 3," or "Object 4;" descriptions will be replaced by object-level descriptions such as "shiny cube;" relations will be replaced by relational concepts in the dataset. In our data generation, we use four object descriptions (one for each object) and three object relation descriptions. Furthermore, we make sure that none of the object-level descriptions can uniquely identify an object, but the global solution is unique.
CLEVR-RPM The CLEVR-RPM dataset is generated based on the following template: "There are 9 objects, ordered in a  $3 \times 3$  grid: row 1 col 1 is a description 1; row 1 col 2 is a description 2; row 1 col 3 is a description 3; row 2 col 1 is a description 4; row 2 col 2 is a description 5; row 2 col 3 is a description 6; row 3 col 1 is a description 7; row 3 col 2 is a description 8; I am missing one object at row 3 col 3. Can you find an object in the scene that can fit there?" Here, descriptions are object-level descriptions such as "shiny cube." Following existing work on Raven's Progressive Matrices [Barrett et al., 2018], we randomly choose the "progression dimensions" from color, shape, material, and size for rows and columns.
C Model
LEFT consists of several modular components in code. The first is a GPT-3.5 [Brown et al., 2020] backed language interpreter, which takes as input queries from each dataset, and outputs first-order logic. The domain initializer takes the generated first-order logic programs and automatically initializes concept embeddings based on the parsed arity. The domain-specific grounding modules extract entity and relational features from the input modality. Finally, the domain-independent executor executes differentiable logic programs with the outputs of the grounding modules. We release our code below.
We use the official data splits released for each dataset, CLEVR [Johnson et al., 2017a], ReferIt3D [Achlioptas et al., 2020], HumanMotionQA [Endo et al., 2023], and Cliport [Shridhar et al., 2022]. The core hyperparameters were set as 128 for concept embedding dimensions, and learning rate taken from prior neuro-symbolic concept learning repositories that we use as baselines. The full details are provided with the code, linked below.
C.3 Compute
We trained with 1 NVIDIA Titan RTX per experiment for all datasets, from an internal cluster.
C.4 Code
We publicly release our code. See our project website for additional details.