Bridging Neural and Symbolic Representations with Transitional Dictionary Learning
Authors: Junyan Cheng
Peter Chin
Venue: ICLR 2024 poster (2024)
Keywords: Unsupervised Learning
Compositional representation
neural-symbolic learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose Transitional Dictionary Learning to learn symbolic knowledge in representation. Experiments on abstract objects show our method largely outperforms unsupervised segmentation baselines and proposed metrics align well with human evaluations.

Abstract: This paper introduces a novel Transitional Dictionary Learning (TDL) framework that can implicitly learn symbolic knowledge, such as visual parts and relations, by reconstructing the input as a combination of parts with implicit relations. We propose a game-theoretic diffusion model to decompose the input into visual parts using the dictionaries learned by the Expectation Maximization (EM) algorithm, implemented as the online prototype clustering, based on the decomposition results. Additionally, two metrics, clustering information gain, and heuristic shape score are proposed to evaluate the model. Experiments are conducted on three abstract compositional visual object datasets, which require the model to utilize the compositionality of data instead of simply exploiting visual features.  Then, three tasks on symbol grounding to predefined classes of parts and relations, as well as transfer learning to unseen classes, followed by a human evaluation, were carried out on these datasets. The results show that the proposed method discovers compositional patterns, which significantly outperforms the state-of-the-art unsupervised part segmentation methods that rely on visual features from pre-trained backbones. Furthermore, the proposed metrics are consistent with human evaluations.

Published as a conference paper at ICLR 2024
BRIDGING NEURAL AND SYMBOLIC REPRESENTA- TIONS WITH TRANSITIONAL DICTIONARY LEARNING
Junyan Cheng
Peter Chin
Thayer School of Engineering
Thayer School of Engineering
Dartmouth College Hanover, NH 03755, USA jc. te.thhdarttuthh. edu
Dartmouth College Hanover, NH 03755, USA
pc@dartmouth. edu
ABSTRACT
This paper introduces novel Transitional Dictionary Learning (TDL. framework that can implicitly learn symbolic knowledge, such visual parts and relations, by reconstructing the input as a combination parts with implicit relations. We propose theoretce diffusion model to decompose the input into visual parts using the dictionaries learned by the Expectation Maximization (EM) algorithm, implemented the online prototype clustering, based on the decomposition results. Additionally, two metrics, clustering information gain, and heuristic shape score are proposed to evaluate the model. Experiments compositional visual object datasets, which require the model to utilize the compo- sitionality of data instead of simply exploiting visual features. Then, three tasks on symbol grounding to predefined classes of parts and relations, as well as transfer learning to unseen classes, followed by a human evaluation, were carried out on these datasets. The results show that the proposed method discovers compositional patterns, which significantly outperforms the state-of-the-art unsupervised part segmentation methods that rely visual features from pre-trained backbones. Furthermore, the proposed metrics are consistent with human evaluations.
conducted on three abstract
INTRODUCTION
In recent years, there has been a desire to incorporate the interpretabilityy compositionality, and logistics of symbolic systems into neural networks (NNs). Existing methods combine them in an ad hoc manner, Dong et al. ( Wang et al. (2019) converts symbolic programs into differentiable forms, Cornelio et al. (023 Segler et al. (2018) introduce symbolic modules to assist NNs, Gupta & Kembhavi (2023) use neural and symbolic modules as building blocks for visual programs. These methods do not truly bring symbolic power to NNs, but simply allow them to work synergically. The essential disparity lies at a low level. NNs use distributed representations, while symbolic systems symbolic representations. This motivates representations, thus combining the two types of intelligence from the ground up.
explore ways bridge neural and symbolic
Cognitive science studies (Taniguchi et al. 2018) have suggested that symbolic representation in the human brain does not appear out of nowhere; rather, there is a gradual transition from neural perception preliminary symbols and eventually to symbolic languages over the course of human evolution, as people observe and interact with their environment. The transitional representation, preliminary symbols, is essential connecting neural and symbolic representation. Taking this concept into account, we attempt to replicate the process of transitional representation arising from neural representation, through unsupervised learning on visual observations in this study, as an exploration of the potential path of unifying neural and symbolic thinking the representation level.
Representation explains how the input is made up of reusable components (Bengio et al., 2013). Taking visual observations as principal components, that illustrate the high- nigh-dimensional statistical features, and symbolic representation uses structural methods, such logic sentences, that explain the visual parts and their connections. A transitional representation should be in between that (1) contains nigh-dimensional details of the input and (2) implies structural information about the semantics of the input.
example, distributed representation is explained by vectors, such

---

Published as a conference paper at ICLR 2024
LineWorld
OmniGlot
ShapeNet5
Figure 1: Decomposing samples from three datasets into visual parts marked with different colors, shallowness represents confidence. Odd columns are input, even columns are decompositions,
We propose a novel Transitional Dictionary Learning (TDL) framework that implicitly learns symbolic knowledge, such visual parts and relations, by reconstructing the input as a combination of parts with implicit relations. With simple fine-tuning that aligns the output with human preference through reinforcement learning and heuristic reward, the model can give human- -iteannitterpretabll decomposition of the input; examples are shown Figure TDL uses an Expectation Maximization (EM) algorithm to iteratively update dictionaries that store hidden representations of symbolic knowledge, through an online prototype clustering on the visual parts decomposed from the inputs by novel same-theorette diffusion model using the current dictionaries.
We suggest two metrics to evaluate the learned representation. Clustering Information Gain, which assesses if the learned dictionary is parsimonious and representative, and heuristic shape score that assesses if the decompositions are in line with human intuition. We conduct unsupervised learning experiments on three abstract compositional visual object datasets, which require the model utilize the compositionality of data instead of simply visual features, and three tasks on symbol grounding to predefined classes of parts and relations, and transfer learning to unseen classes. The results show huge improvements compared to the tate-offthhe-ar part segmentation baselines, which struggle to process abstract objects that lack distinct visual features. We also conduct human evaluations; the results demonstrate significantly improved interpretability of the proposed method and the proposed metrics are consistent with human assessments. Our contributions are concluded as follows.
We propose the unsupervised Transitional Dictionary Learning learn symbolic features in representations with a novel game-theoretic diffusion model and online prototype clustering.
We introduce two metrics, the clustering information gain and the heuristic shape score, to evaluate the learned representation and give evaluation results agreed to human judgment.
We perform experiments, compare our method with state-of-ther unsupervised part segmentation models, and conduct human evaluations for all models and proposed metrics.
Our code and data are available at
RELATED WORK
Neural-Symbolic Learning. Some approaches incorporate symbolic program assist NNs. Segler et al. (2018) used a tree search in learning retrosynthetic routes, Amizadeh et al.) (2020) applied first-order logic answer visual questions, and Young et al. (2019) introduced a symbolic controller to generate repeating visual patterns. However, they are usually tailored to specific tasks. Program synthesis could be more flexible. [nala al. (2020) synthesized multi-agent communication policies, Sun et al.] (202T) searched for autonomous vehicle control programs, and Gupta & Kembhavi (2023) synthesized visual task scripts. Nevertheless, symbolic thinking was not truly incorporated. Thus, differentiable symbolic modules are proposed. [Wang et al. (2019) relaxed an SAT solver as a NN layer, Riegel et al. (2020) made first-order logic propositions differentiable, Dong et al. (2019) used NNs trainable logical functions, Dai et al. (2019) optimized NN with abductive logic, and Goyal ct al. (202T) learned neural production systems of visual entities. However, they overlooked the fact that the disparity between neural and symbolic representations is the source of the problem.
Compositional Representation. Fei-Fei & Perona (205);; Csurka et al. (2004) show early efforts to learn compositionality through the bag of words. Hinton 2021 proposed architecture to learn
2

---

Published as a conference paper at ICLR 2024
part-whole hierarchies, which Garau et al. (2022) implemented using an attention model. Du et al. (2020) employed Energy- energy-Bassed Models (EBMs) to learn compositional parts for image generation. Chen et al. (2020) learned to compose programs from basic blocks. Mendez et al. (2022) learned reusable compositions for lifelong learning. Lake et al. (2015) used Bayesian Program Learning to compose handwritten characters from parts and strokes. [Shanahan al. (2020) used attention to learn relations among objects. |Ma et al. (2019) learned visual concept embeddings and Cao et al. (202I) learned them as prototypes. Kipf et al. (2020) used contrastive learning to embed concepts and implicit relations, and Wu et al. (2022) employed EBMs to embed concepts and relations for zero-shot inference. Du al. (2021) also used EBM to represent concepts. [eCun (2022) introduced an EBM-based framework to learn hierarchical planning. In comparison, we aim to learn the seamless transition between neural and symbolic representations unsupervisedly.
Unsupervised Segmentation. Segmentation extracts structural information from visual inputs.
Lin et al. (2020) used spatial attention to learn scene parsing, |Bear et al. (2020) utilized physical properties, Kim et al. (2020) clustered feature maps, and Van Gansbeke al. (2021) improved it by
contrastive learning. Lou et al. (2022) parsed scene graphs by aligning the image with the dependency graph of a caption. Mielas-Kyriaze
al. (2022) used deep spectral methods segment. The parsed elements in these methods are not reusable; co-part segmentation attempts to address it. (Coliin et al., (2018) learned visual concepts by NMF, Hung et al. (2019); ) Choudhury et al. (202T) learned reusable parts by self-supervised learning. |mmir et al. (0221) extracted concepts from pre-trained vision Transformer, Gao et al. (2021) learned co-parts utilizing motions in videos, Ziegler & Asano (2022) uncovered parts with a Sinkhorn-
clustering, Yu et al. (2022) utilized capsule networks to discover face parts and He et al. (2022) learned parts by hierarchical image generation. However, they rely on concrete visual features and learn visual patterns instead of discovering compositionality like us, thus not working on abstract input, as shown in experiments in Section 5
3 TRANSITIONAL DICTIONARY LEARNING
We first introduce the transitional representation and its optimization target in Section
3.1 then
optimize this target with our TDL framework based on an EM algorithm in Section 3.2 propose the clustering information gain to evaluate the learned representation in Section 3.3 We use
Finally, we
this convention if it is not specified separately: superscript /' denotes i-arity, superscript brackets denotes i-th sample in a dataset, and subscript ;; denotes i-th visual part in sample.
(0) with
3.1 TRANSITIONAL REPRESENTATION
Given visual input E # suppose 2D here for simplicity without loss of generality, can compress it into a dow-dimensional embedding r = fx) E Rd using a NN or other machine learning models that minimizes the reconstruction error by min, e(g(r), x)) with a decoder g. As discussed above, such representations lack interpretability, compositionality, and structural information.
Alternatively, can employ symbolic representation that explicitly identifies structural in- formation. Predicate logic, the dominant and theoretically complete (Newell & Simon, 1976) symbolic representation, expresses the input I as conjunction of logical statements '2 (-() A p-)-) A... A (f(-))) A (((())) A.. that minimizes semantic distance ming ds(2,, x), where pf the i-th logical sentence using a predicate of arity k. the number of arguments. Arguments can be logic variables, constants, or even logical sentences that form high-order sentences.
To simplify the analysis while keeping generality, we assume that an input x linearly composed of visual parts x; by x Σin Iy, where Np is the number parts, although non-linear assumptions exist, such as viewing an image stack of layers or projection from a 3D space. To create meaningful logical representation that is semantically close to x, we begin with the entity mappings I; - Pj. grounding a 1-ary predicate P. D of entities from the dictionary D such as Cat(z,), Tree(x;), Person(z:), in 24. Then, construct relation mappings with higher-ary predicates, taking 2-ary as an example, (z,, 24)) + Ph. where p E D2, such as left.of (I;, 2,)), larger x.),) Finally, we depict the attributes by predicates such as Red(x,)) Length 5cm).
This process is an optimization problem arg maxe P(D]rr, D) where D = (D), "D' D2....} is a dic- tionary collection of different ary predicates. There are major drawbacks, which also led to
3

---

Published as a conference paper at ICLR 2024
the downfall of symbolic AL Firstly, symbol grounding that links predicates to visual components is non non-trivial. Although it can be automated by supervised learning, annotation is expensive and inflexible. Secondly, designing attribute predicates that capture all the details is impossible.
Therefore, we propose a transitional representation, the neural logic variables = (ri, r2,...., ".,, generated by model f(2; 0) with a hidden dictionary parameterized by as D. R is composed of entity vectors r; E Rd and follows the optimization target.
min Σ (((g( 2), + aEplds(o)(R))))) 0), x0)) where R f(x(0,
(I)
0
where R((0) = (r) (1) ب are variables for (0)) in dataset X = {xx1) JE we omit the parameters of decomposition model f and that also need to be optimized in this target for simplicity. minimizes both the reconstruction error of the first "neural" term, where g(
مكك (6, 0) and
g(r)"); 0) is the decoder, and the expected semantic distance of all meaningful concrete dictionaries by the second "symbolic" term. The predicate head 9D can map R( to logic sentences (2 = gD (R)) that maximally preserve the semantics of the input given a concrete dictionary D. ds is an ideal metric that can accurately measure whether two representations express the same semantics. The expectation considers all meaningful dictionaries of an input (e.g., different fonts for the same character), while non-meaningful ones are not considered (e.g., the inputs are dogs, the dictionary for cats). The coefficient adjusts the two goals. In practice, representations that have minimal possible distances from all concrete ones, and then align each by fine-tuning. Transitional representation tackles the second problem above by compressing attributes in embeddings and the first problem with unsupervised learning will be discussed in Section 3.2
train "average" dictionary with transitional
3.2 EXPECTATION MAXIMIZATION EXPRCCATIONNMXXIMIAATTN FOR TRANSITIONAL REPRESENTATION
The first term in Equation |
can be optimized by the following target Kreutz-Delgado et al. 2003
N
Np
Np lrfl
arg min
Σ
(1)
+
3
optimizes the hidden dictionary 9 by minimizing the reconstruction error from visual parts. However, the key challenge comes from the second term. We consider R (r; Nr or its corresponding visual parts, as bag of words for the image x that implicates hidden logical sentences Qg = go (R).
where the optimal 0² arg min EE (dsss((pp ( (R),go(()) (R))) As we only need to consider meaningful dictionaries D = arg min, dg(g;, (R)) [R), x)) that allows semantically equivalent representations of input v, an alternative target for Equation is: ming Σ." (()(R((); ()), (()) (2)( go ())
PP(2 (0) (000) 0) where meaningful logic variables and relations are more likely to appear in the dataset than non-meaningful ones, i.e., reusable and compositional. In other words, the optimal dictionary 6* maximizes the likelihood of the dataset. By regarding (+()) as a visual sentence composed of visual words R(() and dataset X as visual corpus, we optimize the second term in the alternative target via EM algorithm inspired by the Unigram Language Model (ULM) (Kudo 2018) which maximizes the likelihood of the dataset by iteratively updating the dictionaries given decomposed visual parts using current dictionary
We can reasonably assume that (s(x(() go (R()))
Σ log p( (0.0))
arg max 9
The likelihood of the dataset is computed as the summation of the log-likelihood of the logic representations of all sample ((()) from 1 to NA arities by L = D.WI ENA log P(S la: 0). For 1-ary, we follow ULM by log P(Ω) (2c) (z((),
(o((),0) 0) = Σ log P(r for 2-ary, the Markov
assumption for sequential data is not suitable, thus we use a joint probability log P(0 (z(0),))
Nr Nr. log ((r(() ,(()), and the same applies for higher arities.
4

---

Published as a conference paper at ICLR 2024
The optimization targets in Equations 2 and give Transitional Dictionary Learning (TDL) framework. Equation 3 can be optimized by clustering all decomposed visual parts, pairs of parts, etc. The complexity increases exponentially with the arity which is unacceptable despite low-ary is adequate to provide a graph-level representation power, we use techniques like online clustering and random sampling improve the efficiency that discussed later in Section 4.2 Further discussion of the limitations and the broader impacts of the TDL framework can be found in Appendix L
3.3 CLUSTERING INFORMATION GAIN
We wish that the learned predicates are reusable and compositional, the decomposed parts and pairs of the test set should be clustered in few centroids C as possible. Thus, propose Clustering Information Gain (CIG) by comparing the Mean Clustering Error (MCE) the decomposed parts in the test set, marked MCE |E ΣØ=1 position M C Erand, which is lower bound when the decomposed terms are randomly scattered, while the best case of MCE is when the parts match perfectly learned predicates. CIG given by CIG M MCEmmdeel /MC normalized between [o, 1]. See Appendix [ more details.
Np (mincee I|r cll2)/NN\/N with the random decom-
METHOD
Repeat tize steps
Reconstruction Ernor 4a..
Replicote
Izpú
Add noliees
Player
Decomposition
Loss Acten peetory
Tandon Flayer Jn. bobokiigg Insttalizatson
Input boctch
Maska Kiya
wer dmgg
Player Ny
Sieaal parts Leg
Shared-Param
Trarsitiona Inpoantttioo
Pappers fn
Hewory
Clustering Loss Accuinnn
Comperater
Saaperatzr
Practiaaee
Players
8X1 bUKil
Opttmizer
Figure 2: Overview of architecture. The decomposition process takes K steps iteratively refine the generated visual parts. Np models generate in parallel, each generates one part and communicates through "Broadcast". The mapped representations will be stored in memory bank for clustering.
To implement the TDL, need an encoder f R that decomposes the input into R { where each r; decoded by decoder g a mask. Inspired by Wu et al. (2022). we adopt a U-Net-based diffusion model [ong et al. (2021) iteratively refine the generated mask, the encoder downsamples the æ to the embedding r; which upsampled by the decoder as m, The model iterates K steps. Np copies of the model sharing the same parameters, generate Np masks in parallel with each produces one mask. At each step, for model i, the mask m,(tt) generated in the previous step (a random mask for the first step) and other feature maps, such other models' output, are inputted, to produce an updated mask (t + 1). After K steps, the model outputs Np visual parts and the corresponding representation R.
m, into visual part J. m,3, m; E 00, 1 H
To generate multiple meaningful visual parts the same time, we propose a game-theoretic method in Section 4.1 inspired by /eemp et al. (202)) who model PCA competitive game of principal components.
regard the decomposition process as a cooperative game visual parts that converge in K steps to cooperatively reconstruct the input while competing with each other to avoid repetition and so on. Each part is adjusted by a "player", one of the Np copies of the model. With the visual parts generated,
prototype clustering to implement the EM algorithm in Section 4.2 We also introduce a shape score to measure model performance and serve as a reward to tune the unsupervised learned model with reinforcement learning in Section 4.3 Figure 2 shows overview of our method.
5

---

Published as a conference paper at ICLR 2024
4.1 GAME-THEORETIC DECOMPOSITION
A player model adjusts the generated visual parts to maximize the utility modeled by a GT loss
LGT Lee + @iLoverlap + aggrrssureee + a3Lnnom
evaluates the equilibrium state composed of Np generated parts F. (21 22, Np. ). detail:
Reconstruction Error. Litee evaluates the reconstructed input こ = EN
No x; using a combination of
focal loss (Lin et al. 2017) and dice loss (Milletari et al. 2016).
Overlapping Penalty. Loverlop =
>H.w max(0, -)) introduces a competition between players that avoids overlap between parts by penalizing redundant parts.
Resources Penalty. Lresources = =N mar(0, qR - mmal) where qR E R is the quota of player, prevents one player from reconstructing everything while others output empty. The quota restricts one player from having enough resources to output the entire input, thus requiring cooperation.
Np || where nn is the unactivated mask before input into the Sigmoid function, which can simplify the search space of the model accelerate convergence.
L2 Norm. Lnorm Σn
See Appendix D for further details. We follow SMLD (Song et al. 2021 to train a scoring network (m LET LGT = (t);;(t); (,));)000)) 6). where e = {Ø" 3 are prototype dictionaries from I to NA arities that will be discussed in Section 4.2 approximate the gradient of optimal move for each player i that maximizes utility - -Ler. The input state st(t) = (t, (t)) includes the feature map 3 (1) = (o; (4; Exzi ml..)).... that contains the input, the current mask, and the moves of other players (i.e., "broadcast"), and an embedding (4) e(sme()) (L) + en + e(""o(() covers time step (f)m E RRdoms and player index exidd E Rddeo from learned embedding tables, and a predicate embedding
of pred (1)= E P(a (2; (4))r computed for every step.
The move sampled by Langevin dynamics m2(1 (1 + 1) = mm((() + eV m. V (O, 7),t 0, 1,... where is the step size. A loss term LSMLD from the SMLD pa- per can be used to minimize El[[Vm, (t) Ler (t()s(t); 0)][3] to train the scoring network to give good approximations. We apply this term as a regularization form Decomposition Loss Decompooitionn LCT + BLSMLD in Figure]
LoT + V2ez(1), =(1) ~
Further details can be found in the Appendix A
4.2 ONLINE PROTOTYPE CLUSTERING
We cluster R to implement the EM algorithm in Equation 3 As discussed in Section 3. we learn multiple dictionaries for different arities. Each dictionary is composed of prototypes df E R No xd, where Ng. is the dictionary size and de is the dimension of the prototype. We train predicate head for each dictionary; for example, for 1-ary, no (r,) maps neural logic variable r; representation μu E Rdv, for 2-ary, 1 = (r(r.rr)) maps a pair (ri,ry). In our work, mappers are implemented as convolution layers on visual parts and their combinations, e.g., 1 = f2 + xz)-
We perform online clustering during training by maintaining FIFO memory bank M f (M NA where M R R xdy that adds terms j after each training step. Similar to Caron al. 2018 run M-Means for every one a few training steps after the warm-up epochs, in set μa + M¹ for each dictionary, where, taking 1-ary as an example, /² = (n²(), p (2) ..... P..,.," (B)) is the set of 1-ary representations in an input batch of length B. pa has gradients while M' does not. We drop unwanted terms, such as empty ones, and randomly sample, for example, 30% pairs to increase efficiency.
The K-Means output assignments C' for each term in jt', we make pseudo-labels Y for prototypes in the dictionary of by assigning them to their nearest clustering centroids from C². Then we minimize the distance between prototypes and their assigned centroids in latent space by Cross-Entropt (CE)
loss mings, Ti C. (ossssssss(())) (je) d²), Y') where dist is distance metric (e.g. L.2 distance).
The Clustering Loss is the sum of the CE loss for all dictionaries LCtustering = = E/>1 Lgg. We optimize it with the decomposition loss LTDD LCCustering + I. DDecomposition to implement the

---

Published as conference paper at ICLR 2024
TDL framework. More details can be found in the Appendix A.2 a model trained in Line World for all the decomposed parts of the test set in Appendix by tSNE, where that the predicates in the dictionary learned as different clusters, while the baseline, UPD, does not provide predicate information and shows a less organized latent space.
We visualize the latent space of
4.3 REINFORCHMENT LEARNING AND SHAPE SCORE
We employ PPO (Schulman et al. 2017) tune an unsupervised learning model by regarding the decomposition process as an episode. We use a heuristic shape score to evaluate the decomposed part by 3 factors. (I) Continuity Rome: the shape is not segmented and is an integral whole. man(Ac) where Ag is the list of contoured areas for segments in a part. Reont = the part is not segmented. We use findConlours in OpenCV to obtain the segments for 2D data and DBSCAN for 3D after converting to point clouds. (2) Solidity Raolid no holes inside part. Reolid sum(Ac) where Ap the space or area of the part. Riaaidd = if there no hole. (3) Smoothness : the surfaces or contours of the part are smooth. Remooth = Ps where Ps is the perimeter of the smoothed largest contour and po is for the original contour. We apply RDP to smooth 2D data and alpha shape for 3D. The shape score Rs Reont Roolid X Rymooth normalized between and 1. It can also be used to measure model performance. See Appendix for more details.
Reons sum(Ac)
EXPERIMENTS
H
H
Q
H
3
3
3
3
Input
Ours
DFF
SCOP5
UPD
Input
Ours
DFF
SCOPS
UPD
Figure 3: Examples from OmniGlot test set. Our method generates multiple interpretable strokes to reconstruct the input hand-written characters. As a comparison, the baseline methods segment the input into colored parts that are not valid strokes revealing a failure in learning compositionality.
we present our experiment setup to assess whether models can learn meaningful components in abstract objects without supervision, results discussed in Section We then evaluate the learned representations by pre- pre-training the models in unsupervised manner and fine-tuning them for downstream tasks in Section 5.3 Finally, we conduct a human study Section 5.4
In Section 5.1
5.1 EXPERIMENT SETTING
We three abstract compositional visual object datasets as shown Figure where parts cannot be separated by edges, features, colors, etc. Such contiguous shapes can only be decomposed by knowledge of compositionality, thus excluding confoundings. LineWorld generated by the babyARC engine [Wu et al. 2022), consisting of images with to 3 non-overlapping shapes made up of parallel or perpendicular lines. OmniGlot (Lake al. 2015) contains handwritten characters. ShapeNet5 is composed of 3D shapes in 5 categories (bed, chair, table, sofa, lamp) from ShapeNet (Chang et al. 2015) voxelized by binvox (Min) 2004 2019). We replace 2D conv layers with 3D when using this dataset. We create three downstream tasks based them in Section 5.3
We compare three state-of-the art unsupervised part segmentation methods: DFF (Collins et al. 2018) clusters pixels by nog-niettive matrix factorization (NMF) on the activations of the last cony layer: SCOPS (Hung et al., (019) and UPD (Choudhury et al.) 2021) learn to produce a channels heatmap of parts self-supervisedll. The baselines require pre-trained visual backbones. Following
7

---

Published as a conference paper at ICLR 2024
(Choudhury et al. ) VGG19 for 2D data and MedicalNet Chen et al. (019), ResNet- based high-rese reghhresolution 3D medical voxel model, for 3D. We conducted hyperparameter searches for baselines to get their best results. Further details of the setting are provided in the Appendix
5.2
UNSUPERVISED LEARNING OF TRANSITIONAL REPRESENTATION
LineWorld
OmniGlot
ShapeNet5
LW-G
OG-G
IoU
CIG
SP
MAE
CIG
SP
IoU
CIG
SP
IoU
Acc.
loU
AE
97.7
0.9
85.1
DFF
33.1
38.3
36.9
33.3
20.1
19.2
43.1
28.8
42.8
SCO.
35.7
42.4
38.6
38.9
23.1
24.3
46.8
26.4
46.9
UPD
36.3
42.8
42.8
37.4
25.4
22.6
46.2
28.7
48.9
Ours
94.3
58.0
82.6
1.8
68.5
77.6
79.8
54.6
60.1
78.4
74,8
75.9
w/o RL
93.7
57.0
71.9
2.0
65.1
68.0
78.8
52.9
54.4
78.2
74.3
75.1
Table I: Results on unsupervised learning and symbol grounding. SP represents the shape score. MAE and IoU for unsupervised learning are reference-only for comparison with a reference AutoEncoder (AE). Ours and w/o RL are our models with and without RL tuning. We compare DFF (Collins et al. 2018 SCOPS (SCO.) (Hung al. 2019). UPD (Choudhury et 202T.
The results are presented in the first three columns of Table We train Auto-Eneoe Encoders as a reference to see if the generated parts match the input and whether the transitional representation preserves high-dimensional information. The Worl and ShapeNetS inputs are binary, so we use loU for better intuitive. CIG is introduced in Section and the shape score (SP) is discussed in Section 4.3
Our model significantly outperforms the baselines with 58.0, 68.5, 54.6 CIG, and 82.6, 70.6, 60.1 SP in the three datasets, respectively, Even without reinforcement learning, the advantages remain. The low reconstruction error of 94.3 IoU, 1.8 MAE, and 79.8 IoU indicates the preservation of high-dimensional information. This is because the baselines depend concrete visual features such as edges, colors, textures, etc., enabled by pre-trained vision backbones, to identify the boundaries of parts, which are absent in our datasets. For instance, there is explicit color texture difference between strokes in a handwritten character, and seems like contiguous integrity, thus can only be distinguished by the knowledge of strokes, which is learned via discovering compositiona patterns. Figure 3 shows comparison in the OmniGlot test set. See more samples in the Appendix M
5.3
ADAPT TO DOWNSTREAM TASKS
Bed
Lamp
Sofa
Table
IoU
CIG
SP
IoU
CIG
SP
loU
CIG
SP
IoU
CIG
SP
w/ PT
67.3
48.1
52.9
61.1
42.1
49.1
62.2
46.8
45.2
68.3
50.1
54.6
w/o PT
18.1
19.0
13.2
18.3
19.9
14.6
21.5
18.9
19.8
19.9
22.1
17.9
Table 2: Transfer learning for our method on the ShapeGlot setup. "PT" means Pre- Tre-Training.
Symbol Grounding. We design two symbol grounding tasks: LW-G and OG-G, LW-G synthesized with ARC while preserving the shape masks (e.g. lines) and the pair-wise relation annotations (e.g. perpendicular and parallel) from the engine as labels. The goal is to predict the shape masks and classify the pair-wise relations. We aligned the predicted mask with the ground truth by the assignment with minimal overall IoU before computing the metrics. We pre-train the models on Line and add a relation prediction head on the top of baselines while our method directly adapts the 2-ary predicate head for relations classifying. OG-G is subset of OmniGlot with the provided stroke masks as ground truth to predict. We align prediction and ground truth as LW-G. We pre-train models on OmniGlot without OG-G samples. We show examples of LW-G and OG-G in Appendices
8

---

Published as a conference paper at ICLR 2024
and 9 As shown in Table I under LW-G and OG-G, achieve 78.4 IoU, 74.8 Acc. in LW-G and 75.9 IoU in OG-G, outperforming baselines whose relation prediction did not converge due to incorrect segmentations. This demonstrates that the learned transitional representation enables smooth transfer to concrete set of symbols, as hypothesized in Section 3.1
Transfer Learning. Following ShapeGlot (Achlioptas al. 2019 we pre-train with shapes from "chair" and other similar categories, then transfer to 230~550 samples from unseen categories "Bed", "Lamp", "Sofa", "Table". We compare our method with and without pre-training. The results in Table demonstrate that the learned representations are reusable and effectively generalized to unseen classes. Without pre-traie gre-training, the samples for each class were not sufficient to converge.
5.4
HUMAN EVALUATION
Opinians on the decomposition results
Our DFF SCOPS
2.2
CG
SP
0.5
5400000 1af
0.4
1.0
accopnattes
Moratodw
Figure 4: Left: Results of the human evaluation. Right: The qualitative scores compared to metrics.
Human Interpretability. We conduct human evaluation to evaluate our method and the baselines by humans. We randomly selected 500 samples from the OmniGlot test set and the decomposition results for each method. We use the Google Vertex AI (Google 202[) data labeling service to evaluate image annotation task with three annotators. Annotators are given decomposed samples and asked to provide one of four opinions, examples of which can be found in Appendix K as outlined an instruction that must be read before the task begins. The 2000 samples are shuffled and then randomly assigned to the annotators. The results in Figure show a much better interpretability of our method, while 65% of the baseline results are not considered strokes.
the results
Interpretability vs. Metrics. We further train more models, in addition to the models in Section 5.2. get near-even distributed SP and CIG by early stop. We then conduct human evaluations in the same way above. We assign points for each sample by Non-strokes 0, Unnatural: 1, Acceptables 3, Good: 5, then average them as scores for each model. And compare scores with the metrics of each model in Figure 4 right, which shows that SP and CIG are positively correlated with human interpretability, which can be used as reliable predictors of interpretability.
CONCLUSION
This paper presents the TDL framework, which uses an EM algorithm to learn neural-symbolic transitional representation that incorporates structural information into representations. We introduce game-theoretic diffusion model with online prototype clustering to implement TDL and assess by proposed metries, clustering information gain, and shape score. We evaluate our method on three abstract compositional visual object datasets, using unsupervised learning, downstream task experiments, and human assessments. Our results demonstrate that our method largely outperforms existing unsupervised part segmentation methods, which rely visual features instead of discovering compositionality. Furthermore, our proposed metrics are in agreement with human judgment. We believe that work can help bridge the gap between neural and symbolic intelligence.

---

Published as a conference paper at ICLR 2024
7 REPRODUCIBILITY STATEMENT
To guarantee the reproducibility and completeness of this paper, we provide the full details of our model architecture and implementation in the Appendix A Coontans information about the generation or preprocessing of samples for each dataset and the split used in each experiment. Appendix contains our settings for hyperparameter search and our hardware platform information. The tricks we used to calculate the GT loss are included in Appendix B We also make our code and data publicly available for readers to reproduce our work.
REFERENCES
Panos Achlioptas, Judy Fan, Robert Hawkins, Noah Goodman, and Leonidas J Guibas. Shapeglot: Learning language for shape differentiation. Proceedings of the IEEE/CVE International 2019.
Conference on Computer Vision, pp.
Rakesh Agrawal, Tomasz Imielinski, and Arun Swami. Mining association rules between sets of items in large databases. In Proceedings of the 1993 ACM SIGMOD international conference on Management of data, pp. 207-216, 1993.
Amazon. Amazon mechanical turk. https//www..murk.com// 2005.
Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. arXiv preprint arXiv:2 2(3):4, 2021.
Saeed Amizadeh, Hamid Palangi, Alex Polozov, Yichen Huang, and Kazuhito Koishida. Neuro- symbolic visual reasoning: Disentangling. In International Conference on Machine Learning, pp. 279-290 PMLR, 2020.
Daniel Bear, Chaofei Fan, Damian Mrowca, Yunzhu Li, Seth Alter, Aran Nayebi, Jeremy Schwartz, Li Fei-Fei, Jiajun Wu, Josh Tenenbaum, al. Learning physical graph representations from visual scenes. Advances in Neural Information Processing Systems, 33:6027-6039, 2020.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions Pattern Analysis and Machine Intelligence, 3798))1188888 2013. doi: 10.11099/PAM..2013.560
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL hhttps://wwwt com/ Software available from wandb.com.
Kaidi Cao, Maria Brbic, and Jure Leskovec. Concept learners for few-shot learning. In International Conference on Learning Representations, 2021. URL httpppss//penneeeeew..eeeete ffott/fomm?
1d==JIJF33-L220
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In Proceedings of the European conference computer vision (ECCV), pp. 132-149, 2018
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-is 3d model repository. arXiv preprint 1121v:151..03012, 2015. 2015.
Sihong Chen, Kai Ma, and Yefeng Zheng. Med3d: Transfer learning for 3d medical image analysis.
arXiv preprint arXiv: 091400004522.. 2019.
Xinyun Chen, Chen Liang Adams Wei Yu, Denny Zhou, Dawn Song. and Quoc V. Le. Neural symbolic reader: Scalable integration of distributed and symbolic representations for reading comprehension. In International Conference on Learning Representations, 2020. URL https
//openrevient net./ feeet//frrmmhooryyy1ii
Junyan Cheng, lordanis Fostiropoulos, and Barry Bochm. Gn-transformer: Fusing sequence and graph representation for improved code summarization, 2021a.
10

---

Published as a conference paper at ICLR 2024
Junyan Cheng, lordanis Fostiropoulos, and Barry Boehm. Graph conditioned sparse sparse attention for improved source code understanding, 2021b.
Junyan Cheng, Tordanis Fostiropoulos, Barry Boehm, and Mohammad Soleymani. Multimodal the
phased transformer for sentiment analysis. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing. pp. 2447-2458, Online and Punta Cana, Dominican Republic, November 2021c. Association for Computational Linguistics. doi: 10. 18653/v 1/2021. umalp-main.189 URL https::ppp.... :/ac/ http//cclantooooooooooooooooooo org/2021 emn p- i n. 189
Subhabrata Choudhury, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Unsupervised part discovery from contrastive reconstruction. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https:// wtw.:/uueee/toututteeemm openreview net forum idd HXQQ I
Edo Collins, Radhakrishna Achanta, and Sabine Susstrunk. Deep feature factorization for concept discovery. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 336-352, 2018.
Cristina Cornelo,, Jan Stuehmer, Shell Xu Hu, and Timothy Hospedales. Learning where and when to reason in neuro-symbolic inference. In The Eleventh International Conference on Learning Representations, 2023. URL hattps://openreetee
Gabriella Csurka, Christopher Dance, Lixin Fan, Jutta Willamowski, and Cédric Bray. Visual categorization with bags of keypoints. In Workshop on statistical learning in computer vision, ECCV, volume I, pp. 1-2. Prague, 2004.
Wang-Zhou Dai, Qiuling Xu, Yang Yu, and Zhi-Hua Zhou. Bridging machine learning and logical reasoning by abductive learning, Advances in Neural Information Processing Systems, 32, 2019.
Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou. Neural logic machines. In International Conference on Learning Representations, 2019. URL https:// openreview net/ Bttr:/rrrmmdd-1xy--bbeaa-
Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 6637-6647. Curran Asso- ciates, Inc., 2020. URL neurips. /cppaeer...p...e.eeeeeeeeeeee.ee files/paper/ htt.:///SSS..aaaaaaaaafafaaaaaaa
pddf
Yilun Du, Shuang Li, Yash Sharma, Josh Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. Advances in Neural Information Processing Systems, 34: 15608-15620, 2021.
William Falcon and The PyTorch Lightning team. PyTorch Lightning, March 2019. URL https //github. wwt..o///.ggggnnnggg/naaa..
Li Fei-Fei and Pietro Perona. A bayesian hierarchical model for learning natural scene categories. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05)) volume 2, pp. 524-531. IEEE, 2005.
Qingzhe Gao, Bin Wang, Libin Liu, and Baoquan Chen. Unsupervised co-part segmentation through assembly. In International Conference on Machine Learning, pp. 3576-3586. PMLR, 2021.
Nicola Garau, Niccoló Bisagno, Zeno Sambugaro, and Nicola Conci. Interpretable whot- hierarchies and conceptual relationships in neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 33689-13698, 2022.
Ian Gemp, Brian McWilliams, Claire Vernade, and Thore Graepel. Eigengame: (PCA) as a nash equilibrium. In International Conference on Learning Representations, 2021. URL https!
//openreview: net./ fo
Google. Vertex ai data htt.://.w.....oooooooooco/////on 2021.
11

---

Published as a conference paper at ICLR 2024
Anirudh Goyal, Aniket Didolkar, Nan Rosemary Ke, Charles Blundell, Philippe Beaudoin, Nicolas Heess, Michael C Mozer, and Yoshua Bengio. Neural production systems. Advances in Neural Information Processing Systems, 34:25673-2566 2021.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14953-14962, 2023.
Xingzhe He, Bastian Wandt, and Helge Rhodin. Ganseg: Learning segment by unsupervised hierarchical image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1225-1235, June 2022.
Geoffrey Hinton. How to represent part-whole hierarchies in neural network, 2021.
Wei-Chih Hung, Varun Jampani, Sifei Liu, Pavlo Molchanov, Ming-Hsuar Yang, and Jan Kautz. Scops: Self-supervised co-part segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 869-878, 2019.
Jeevana Priya Inala, Yichen Yang, James Paulos, Yewen Pu, Osbert Bastani, Vijay Kumar, Martin Rinard, and Armando Solar-Lezama. Neurosymbolic transformers for multi-agent communication. Advances in Neural Information Processing Systems, 33:13597-13308, 2020.
Wonjik Kim, Asako Kanezaki, and Masayuki Tanaka. Unsupervised learning of image segmentation based on differentiable feature clustering. IEEE Transactions on Image Processing, 29:8055-8068, 2020.
Thomas Kipf, Elise der Pol, and Max Welling. Contrastive learning of structured world models. In International Conference on Learning Representations, 2020. URL net / Eorum? Fid H1gax6VtDB
Kenneth Kreutz-Delgado, Joseph F. Murray, Bhaskar D. Rao, Kjersti Engan, Te-Won Lee, and Terrence J. Sejnowski. Dictionary Learning Algorithms for Sparse Representation. Neural
Computation, 15(2):349-396 02 2003. ISSN 0899-7667. doi: 101116208997600772555599
URL ht..htts:://ee /ttttps://doi.or//10000 11
Taku Kudo. Subword regularization: Improving neural network translation models with mul- tiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 66-75, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18865/11/P18-10077 URL
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 130062666):1333--338,, 1332-1338, 2015.
Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-21 Open Review, 62, 2022.
Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, and Daniel Haziza. xformers: A modular and hackable transformer modelling library. https://githann com/ facebookresearrchh httperrcchx/foroe xformers 2022.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár, Focal loss for dense object detection. In Proceedings of the IEEE international conference computer vision, pp. 2980-2988, 2017.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. arXiv preprint 2tt:::20010024000 2020.
Chao Lou, Wenjuan Han, Yuhuan Lin, and Zilong Zheng. Unsupervised Dision-languae parsing: Seamlessly bridging visual scene graphs with language structures via dependency relationships. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15607-15616, 2022.
12

---

Published as a conference paper at ICLR 2024
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The neuro- symbolic concept leamer: Interpreting scenes, words, and sentences from natural supervision. In Internationaal Conference Learning Representations, 2019. URL httpsss//ppeeeeiiee net/ d= hRctm
Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8364-8372 2022.
Jorge Mendez, Harm van Seijen, and ERIC EATON. Modular lifelong reinforcement learning via neural composition. In International Conference on Learning Representations, 2022. URL
https://openrest
net/ / forummid==xmmmddsss
Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference 3D vision (3DV), pp. 565-571. leee, 2016.
Patrick
Min.
binvox.
http://www www.iicciiiii ctw.riiiimi...mm/iiiiai..... com/binvox or
https: / www. . gwwgle.oooooooooooooooooooo.o ctm./e....omaeeooeoeee.eeeeeeeee wwt.om/saarchhgbbbaeaaa 2004
2004 2019. Accessed: yyyy-mm-dd.
Allen Newell and Herbert A. Simon. Computer science empirical inquiry: Symbols and search. Commun. ACM, 1333)111111266 mar 1976. ISSN 0001-0782. doi: 10.1 2..114/360018.360000 URL
https::/doff orgg / 10. 1111/1111111111 360022
Fakir S. Nooruddin and Greg Turk. Simplification and repair of polygonal models using volumetric techniques. IEEE Transactions on Visualization and Computer Graphics, 9(2):191-205, 2003.
Augustus Odena, Vincent Dumoulin, and Chris Olah. Deconvolution and checkerboard arti- facts. Distill, 2016. doi: (0.23915/distill..0000... URL http:///issii1.ppuubooo0
deconv-cteckerboarc
OpenAl. Ppo2. http://hhhhclloleooococcccooo/as
2018-2021. 20112
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PMLR, 2021.
Ryan Riegel, Alexander Gray, Francois Luus, Naweed Khan, Ndivhuwo Makondo, Ismail Yunus Akhalwaya, Haifeng Qian, Ronald Fagin, Francisco Barahona, Udit Sharma, et al. Logical neural networks. arXiv preprint 200iv:2006. 13155, 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.06447, 2017.
Marwin HS Segler, Mike Preuss, and Mark Waller. Planning chemical syntheses with deep neural networks and symbolic ai. Nature, 555(7698):604 610, 2018.
Murray Shanahan, Kyriacos Nikiforou, Antonia Creswell, Christos Kaplanis, David Barrett, and Marta Garnelo. An explicitly relational neural network architecture. In International Conference on Machine Learning, pp. 8593-8603, PMLR, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference Learning Representations, 2021. htLps::::/ httpt::::///ppeeeviee.... new.e/wwweeeeeeeeeeeeeeeeeeeeeee wwt.w/wwwwww.wwtwwnntee forum?
1.d=PXTIG12RRHH
Jiankai Sun, Hao Sun, Tian Han, and Bolei Zhou. Neuro-symbolic program search for autonomous driving decision module design. In Conference Robot Learning, pp. 21-30. PMLR, 2021.
13

---

Published as a conference paper at ICLR 2024
Tadahiro Taniguchi, Emre Ugur, Matej Hoffmann, Lorenzo Jamone, Takayuki Nagai, Benjamin Rosman, Toshihiko Matsuka, Naoto Iwahashi, Erhan Oztop, Justus Piater, et al. Symbol emergence in cognitive developmental systems: survey. IEEE transactions Cognitive and Developmental Systems, 1(4):494-516, 2018
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool. Unsupervised semantic segmentation by contrasting object mask proposals. Proceedings of the IEEE/CVF 2021.
International Conference on Computer Vision, pp.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Po-Wei Wang, Priya Donti, Bryan Wilder, and Zico Kolter. Satnet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver. In International Conference on Machine Learning, pp. 6545-6554. PMLR, 2019.
Tailin Wu, Megan Tjandrasuwita. Zhengxuan Wu, Xuelin Yang, Kevin Liu, Rok Sosic, and Jure Leskovec. Zeroc: A neuro-symbolic model for zero-shot concept recognition and acquisition at inference time. Advances Neural Information Processing Systems, 55:9828-9840 2022.
Hong-Ming Yang. Xu-Yao Yao Zhang, Fei Yin, and Cheng-Lin Liu. Robust classification with convolu- tional prototype learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3474-3482 2018.
Halley Young, Osbert Bastani, and Mayur Naik. Learning neurosymbolic generative models via program synthesis. In International Conference on Machine Learning, pp. 7144-7153, PMLR. 2019.
Chang Yu, Xiangyu Zhu, Xiaomei Zhang, Zidu Wang, Zhaoxiang Zhang, and Zhen Lei. Hp-capsule: Unsupervised face part discovery by hicrarchical parsing capsule network. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 4032-4041, June 2022.
Adrian Ziegler and Yuki M Asano. Self-supervised learning of object parts for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14502-14511, 2022.
14

---

Published as a conference paper at ICLR 2024
A
ARCHITECTURE DETAILS
Predicate Head
Repeat time steps
LDM Encoder
Attention
LDM Decoder
U-Net Decoder
U-Net Encoder
(Actor)
Data flow
Attention
Critic
(U-Net) Attention
Predict value
Nornal
Context (Other players)
Itolic
Diflusion model
Predicate head
LDM
Atlention
Optinal module
Figure 5: Illustration of the model architecture. Blocks with italic font are optional optimizations. Different color marks the information flow of the different modules.
We present the details of our model here. The architecture is depicted in Figure which includes two core components, a diffusion model, and predicate head, as well as three optional optimizations: Latent diffusion model (LDM) to improve efficiency, an attention mechanism to incorporate context information such the moves of other players, and PPO tune the model. We assume single 2D input by default in this section for simplicity, and is easy to extend to batch cases.
A.1 DIFFUSION MODEL
Algorithm Decompose an input
Require: an scoring model fs(:;, 6, Φ), decoder fy IRd HxxCC
map representation visual
part
Require: an input æ E "xxxx number players Np, time steps K
1: Randomly initialize variables of Np players R(O)) E Nxxd
2: Initialize player id embedding exid RR. x.xadam and time embedding time E T Xdema
3: for {1..... K} do
4:
(t) = ((((t), where (t) E RNXH*WXC
> visual parts of N players (f)) (f))
5: Vj E (1,-..,NP), (t) = (t); Exts
#x(t); Expioc 6: Vj E (r),.,,N),,,,g = =chrt + +efime (t) + GetPredi
see A.2. for
GetPrediEmb
7: R(t + (()))(())t())))))...
8: end for
9: return R(K)
The algorithm Iillustrates how the model decomposes an input. Notice that there are a few differences between Section where the scoring model also incorporates the prototype dictionaries. Our model is based on SMLD (Song et al. [021)) SMLD adds dense layer between each convolution layer U-Net, which introduces
time-step embedding !iime Rdems by adding the output this dense layer to the output of the convolutional layer. At each time step 1,, for player j. the input state (t) composed of feature maps v (t) E H xCinp and the embedding e(t) E gdems to produce updated decomposed composition z, (t 1).
A.2 PREDICATE HEAD
The predicate head embeds decomposed parts into latent space and clusters them to learn dictionaries for predicates. The head f for i-ary predicates implemented with shared mapper I mapper for
15

---

Published as a conference paper at ICLR 2024
all arities implemented convolution layers with global pooling, mapping part 2, R XxxC to tensor Ty IRdmapper and an arity arity-specific linear layer Q' IR [ddmapper Xdy map the тÿ to embedding My. In our work, we apply the multi multi-prototype trick (Yang et al. 2018), which means that we apply K prototypes for each i-ary cluster, resulting in proti-pr dictionary 6' E RK XN xdy
Algorithm 2 Compute clustering error for and 2-ary predicates
Require: variables R E Fx prototypes o¹ IR IR xd, x² E R.
Require: memory banks M 1 E R 3 Xdy, .M E 1R
Require: mappers f. ff R + 1R decoder fg RR + [=
Require: distance metric for two vector sequences length n1 and ng: d R" x R=2xd H
R
filter out unwanted visual parts D random sample NJ items 3: P filter((x" + x², Yz" x z), where P E R XXXWXC
1: 7 filter(Ss ( where F E RR'XHxWxx
2: / sample(ff) (fl N,), where nt E xdd
pair representations D. random sample N items
4: 112 (omple([2(T))) N2), where p²² E IRR N2dd
5:
6: function ASSIGN(C .6))
C = EDeseNt. 1 o}H((C==k))
LOSSENS
7:
#|C)===) Y0 < K N
P Centroids for i-ary cluster
dg = Li.Dist(C,,,, dE E N.NNN D LI Distance from centroids to permutations
8:
of assignments
9:
while k: < Ng: do
10:
(rowe, col) argmin(ds)
11:
Y = col
dg|row, :| tinf
13:
dgli,co = +inf
end while
15: return y
16:
end function
17:
18: function GETLOSS(I Mi 3*)
19:
C = K-Meanss("' (Jr: M'),, O = -C (...,,,)
Y = Assign(C* 6¹), Y' E N NNN,N
20:
21: dist d(p', 6¹), dist' E R xN,
22: return CrossEntron Loss Yi)
23:
end function
24: return GelLossly M M GetLoss( M2
In each time step t, for player j, the predicate head accepts the current decomposition (t) as input and outputs the predicate embedding and cooperator map. After K time steps, given the representation R, the predicate head computes the clustering error by Algorithm [ We further explore Higher-Order Logic (HOL) predicate, attention, as h = Enys P(r). rrt and HOL pairs ni can be constructed with ht in the same way as 2-ary predicates by summing up the corresponding image parts.
HOL representation is given by dot-product
A.2.1 PREDICATE EMBEDDING
For a player j. the predicate embedding computed as (tred(t) EvEoI P(olaa, (t))o where ((rlz,())) x (dist gf (t)), dist is a distance metric, we use L. distance by default in this work and df (t) = Q fmapper (I, (t)). It represents the potential l-ary predicate of the current decompo- sition result. If using a higher-ordee predicate, we sum of pred (t) with Eo P(alhy(t))o where P(a)hy(()) x -dist(o, d (t)), o Q Smapper (Fg (hy(()) and h²(l) Σxes P(ra, rR)TK where P(ry,rk) rk) ryrk, the dot-product distance, to model relations grouped players.
16

---

Published as a conference paper at ICLR 2024
A.3 LATENT DIFFUSION MODEL
LDM RRmbaac et al. 2022) uses an Auto-Encoder to encode an input I E RHxWxC into compressed input x' E RR "D''WWXC" where H' < H and W < W, and input the compressed input x' instead of the original input x into the diffusion model, which outputs the composition Ty R x'''w'xx xC' latent space, then uses the decoder decompress it into the original pixel space in our work, and we keep the skip connection that inputs the downsampled middle results from the encoder to the decoder.
Ty "XXxxC, We use U-Net-based
A.4 ATTENTION LAYERS
We introduce optional transformer blocks (Vaswani et al., 2017) between the convolution layers in U-Net, similar to CLIP (Radford et al. 2021). There are two types of transformer blocks that have been used in our methods, self-attention, and cross-attention, self-attention is only used in the encoder part when using an LDM since there is no context, and all other places use the cross- attention. For tross-attention, the context the set of the mapped representation of the competitors
{7I, 7j-,, ...mm for player j mapped by fmopper introduced in Section A.2 We utilize memory-efficient attention in xFormers (Lefaudeux et al. 2022) in our implementation.
Attention also gives powerful tool for incorporating multimodal information (Cheng et al. 2021c) and inductive biases (Cheng et al. 2021a the context may come from other modalities, similar to Rombach et al. (202)). ) and in multi-agent case, the context could come from other agents.
A.5
PPO AND ACTOR-CRITIC
PPO (Schulman et al. 2017) uses an Actor- Critic framework to train the agent, we apply a shared encoder actor and critic that the U-Net encoder in the diffusion model is shared, and we train two decoders for the actor and critic, respectively. The actor samples a move with a Bernoulli distribution, where the probability is given by the diffusion model output, on each pixel as mask. A reward function rating on this mask by the loss and the shape score. And the critic is trained to predict the reward given state. We follow the PPO algorithm with the implementation of PPO2 (OpenAI 2018-202T). The model is updated every few steps of sampling. A small buffer that saves actions, states, following states, and other useful information is maintained and retrieved iteratively when the model is updated.
B DATASET DETAILS
We present statistical information about the data we used, the division the training, testing, and development sets, and the details of how generated and pre-processed the datasets.
LineWorld. We employ the ARC engine (Wu et al. 2022) to generate the LineWorld dataset. This dataset consists of objects made up of lines, which are related to each other in terms of parallelism and perpendicularity. Each sample is an image containing between one and three "Lshape", "Tshape", "Eshape", "Rectangle", "Hshape", "Cshape", "Ashape", and "Fshape" objects, each of which may have a different size and one of three randomly selected colors. The shapes are non-overlapping and placed randomly white background. In total, we synthesize 50000 samples, which are divided into 1:1:1 splits for training, development, and testing.
Line Www.d-orronddeeeeoo (LW-G). We used the ARC engine to synthesize the LW-G dataset. The objects in LW-G are composed of lines as a basic concept with four relations to each other:
Parallel.
The lines parallel.
The two lines are vertical, with an endpoint one line attached to the middle of another.
VerticalMid.
VerticalEdge.
The two lines are vertical, with an endpoint of one line attached to an endpoint of
another.
17

---

Published as a conference paper at ICLR 2024
VerticalSepa.
The two lines are vertical, but the endpoint of one line is not attached to another.
Each object is composed of a pattern. Apart from the 4 relationships as the basic pattern that simply samples two random lines following the 4 relations, we design 7 more complex patterns:
Patterns pattern
Sampling process
Sample line in a random position, sample the second line that "VerticalMid" to the first line, then sample third line "VerticalEdge" to the first Sample one more line that "VerticalEdge" to the first line in a "Fpattern" with one endpoint attached to the unattached endpoint of the first line Sample one more line that is "Parallel" to the first line in a "Fpattern", the line can be anywhere that does not overlap with the first line
E pattern
A pattern
C pattern
Sample one line in random position, sample the second line that "VerticalEdge" to the first line, then sample a third line "VerticalEdge" to the first but attach to another endpoint
H pattern
Sample one line in a random position, sample the second line that "VerticalMid" to the first line, then sample a third line "Parallel" to the first Sample one more line that "VerticalEdge" to the second line in a "Fpattemn the unattached endpoint
P pattern
Rect
Sample one more line that "VerticalEdgee" to the second line in "Cpattern" the unattached endpoint
The length and direction of the lines randomly determined and the color of each line is randomly selected from two available colors. This implies that an "F pattern" does not necessarily have to be in the form of "F" the two parallel lines may have different directions and lengths, and this true for all other patterns as well. Each sample is composed of an image of one object, a list of concepts (i.e., lines) where each concept is represented by a mask that points out this concept in the image, and a list of relation tuples between the concepts (e.g. (linel, line2, parallel). In total, we generated 7000 samples, with a 5:1:1 split for train, dev, and test sets.
We assess concept prediction in this manner: concepts and 2-2 connections between them, forming a complete graph Gc. We then determine whether G is included in Gc. We assign c to the closest candidates with the least Intersection over Union (IoU) and calculate the mean IoU. Subsequently, we calculate the top-1 accuracy of the predicted relation based on the assignment of nodes.
Let us assume that the model provides k potential
Our model reads the relation between two parts directly from the two-ary predicates. For baselines, we added relation prediction head that takes two parts of the heatmap input and predicts the relation between them. This prediction head has similar structure to our composition mapper, which is used to embed the outputted composition from each player for clustering. The parts are mapped first, and then a classifier is used to predict the relation.
OmniGlot. The OmniGlot dataset was collected by Lake et al. (2015) using Amazon's 's Mechanical Turk (Amazon 2005). They recruited participants to draw 1623 characters from 50 different alphabets, each of which was drawn 20 times by different people. Each sample was composed of multiple stroke sequences of [ar, y. t] coordinates that documented the strokes used to create the character. We employ the program from Lake et al. (2015) to transform the stroke sequences into images. We allocate 24000, 1500, and 1500 images for the training, testing, and development sets, respectively, and the remaining images are used for the OG-G dataset.
HomiGlot-Grrondiin (OG-G). OG-G is composed of the remaining 5811 samples, apart from the image of the character, each sample also contains set of images of the strokes that are converted from the sequence of each stroke that composed the character as the ground truth. We use 4311, 750, and 750 samples for train, test, and dev sets.
ShapeNet5. ShapeNet5 consisted of all 20938 shapes of 5 categories (bed, chair, table, sofa, lamp), suggested by AAchiiopas et al. (2019) that they are composed of shared basic elements, from ShapeNetCore v2 (Chang et al. 2015), updated version of the core ShapeNet dataset. There
18

---

Published as a conference paper at ICLR 2024
many ways represent 3D data including point clouds, meshes, voxels, Signed Distance Fields (SDF), and octrees. In order to make 3D shapes directly applied to the same architecture with other datasets, choose voxelize the shapes; thus, can handle them by simply replacing 2D convolutions with 3D. We use the binvox library (Min) 2004 2019, Nooruddin & Turk 2003) to voxelize shapes into solid voxels (i.e., the interiors of the shapes are filled). We applied 15938, 2500, and 2500 samples for train, test, and dev sets.
ShapeGlot Transfer Learning. This transfer learning dataset consists of a pre-training set and fine-tuning sets. The pretraining set covers 1470 samples from 5 categories (chair, bench, cabinet, bookshelf, bathtub) that we regard as having similar basic compositions. Four fine-tuning sets correspond to four categories that share similar basic elements with the pretraining set: bed (233), lamp (532), sofa (550), and table (580), number of samples for each category provided brackets. We voxelize the samples the same way with ShapeNet5.
C HYPER-PARAMETER SEARCH
Params
Distribution
Params
Distribution
lr
{le 3, 2e - 3, 5e - 4} ths
(00.1,, 0.5; 0.05) or None
K
U33, 9;
quota
U(8 32; 4)
Okoverlap
(00.1,0.25)
demb
(64,122,256,51)
Oresources
(0.055, 0.2)
deampter dmapper
(32.64) { (128,256,512)
Veluster
USe - 2e -
12,5,5,10,15,555
Table 3: Hyper- search distributions.
We use Weights & Biases Sweep (Biewald) 2020) to perform hyperparameter searches for our method. Table 3 shows the distribution of the empirically significant parameters in hyper-param search. O'veelap and Gresources are 01 and az in Equation 4 Ua, is a discrete uniform distribution with step size of between and b, while U(a, b) uniform distribution between and b. We use a combination of random search and grid search to explore the search space, with random search used identify good traces, and then, with the help of the visualization tool provided by Sweep, get a smaller search space for a finer grid search. The final good range determined by the random search can vary for different experiments: however, the distribution given by Table provides the common initial range that empirically likely covers the optimal sets to explore.
We conducted our experiments on our internal clusters, and major workload has the following configuration: six Quadro RTX 5000 GPUs and one Quadro RTX 8000 GPU, along with an Intel (R) Xeon (R) Silver 4214R CPU @ 2.40GHz and 386 GB RAM. We employed PyTorch Lightning FFalcon & The PoTor Lightning team 2019) for parallel training.
D LATENT SPACE VISUALIZATION
Figure 6: t-SNE for parts segmented by UPD Choudhury al. 2021 (left), and the latent space of (mid) and 2-ary (right) embeddings of parts and pairs, colored by nearest predicates, decomposed by our method in the Wirld test set. UPD provides no such predicate information thus not colored.
19

---

Published as a conference paper at ICLR 2024
E
GT LOSS DETAILS
I I I
IIIE
C 1.
T
H
weak-overlapping
Chessboard- peessboard-overLappiig overlapping
Non-inteerpretable
Instance separation
Figure 7: Unwanted cases shown during mechanism design. The first shows pairs of input (left) and reconstructed result (right), and the remaining rows are the outputs of players. Another common cases not shown here are all players output the same output and one player output while all others are empty.
There some important tricks apply apart from only using the GT loss avoid some unwanted outputs as shown in Figure 7 The weak-overlapping the mechanisms.
a tricky way learned by the model to bypass means that each player produces weak copy of the input æy 0 < c < 1, it fully meets the requirements of the GT loss. We solve it with soft step function trick, which computing the overlapping loss by max ΣS NN Step(zz; ths) - Step(z;; ths)), where Step is point-wise step function that assigns to position the value is above a threshold thy and O otherwise. A soft step function can be implemented with Heaviside function. With a step function, the overlapping loss becomes more sensitive, and a small value in position will be regarded as an occupation. And the threshold controls the sensitivity. Thus, the model cannot cheat with weak overlapping and be more careful when assigning values.
where
Another undesirable situation chessboard overlapping, where players avoid overlap by outputting pixels with intervals. This is typically caused by using deconvolutional upsampling (Odena et al.) 2016) and can solved by replacing with bilinear interpolation. The non-interpretabl shape occurs in a base model without a predicate head. By applying predicate clustering and other techniques that encourage interpretability, such cases can be largely reduced, In the instance separation case, the model learns unwanted good case that gives segmentation of instances. This case is caused by the given dictionary size being too large, thus the model can memorize all shapes.
F ABLATION STUDY ON MODEL. ARCHITECTURE
Line World
OmniGlot
loU
CIG
SP
MAE
CIG
SP
Base
91.3
26.3
28.4
2.5
31.8
23.8
Cluster
92.3
54.6
68.9
2.4
61.7
64.8
2 2220 2
2
+ HOL
92.7
56.9
71.5
2.0
64.7
67.5
+ PPO
91.9
57.1
82.6
2.0
67.2
77.4
+ Attn.
94.9
56.2
70.8
1.8
63.4
66.8
+ LDM
91.7
53.6
66.9
3.2
59.9
62.8
Full model
94.3
58.0
82.6
1.8
68.5
77.6
Table Ablation study results.
means repeating the above row.
20

---

Published as a conference paper at ICLR 2024
We perform ablation study of the proposed predicate clustering method, PPO tuning, and opti- mizations in Line World and OmniGlot datasets. The results are listed in Table 4
"Base" means the model trained with only decomposition loss. Without clustering, the model gives an arbitrary decomposition that reconstructs the imput while meeting the game mechanisms, which generate parts with diverse near-random shapes, thus having low CIG and SP.
"+ Cluster" means adding a clustering loss to the base model. With cluster loss, the model does dictionary learning, which significantly improves the CIG and SP, since the model learns to find common elements to represent the data. also shows that learning an efficient dictionary itself also results in simpler and more natural shapes. Although not sufficient learn the human interpretable shapes.
"+ HOL" adds the higher-order predicate optimization the model with clustering, and shows marginal improvement, which may be due to the low complexity of the shapes in our datasets, which do contain too complex relationships that need to be depicted with higher-order predicates. And better way of representing higher-order predicates may also lead to better results.
"+ PPO" adds PPO tuning with heuristic reward the model with clustering, and it clearly improves the SP in both datasets due to the introduction of an explicit bias that guides the model to learn more natural shapes, which shows the importance of feedback and environmental interactions.
"+ Attn." adds attention layers to the model with clustering; it also gives marginal improvement which may be due to the relatively limited complexity our dataset. Moreover, as discussed earlier, a better case for cross-attention is multimodal learning. And we do not test self-attention, which we regard should be applied to larger datasets.
"+ LDM" uses a latent diffusion in the model with clustering, the result shows that the use of LDM has only limited harm to the model performance; compared to efficiency improvement, the downside quite acceptable.
In conclusion, the findings demonstrate that dictionary learning critical factor in learning transitional representation, while PPO can effectively enhance the representation. HOL and attention offer minor improvements in our datasets; however, they may be more beneficial in more complex dataset and with better higher-order representation. L.M can improve model efficiency with minimal impact on performance, which is essential for scaling to larger inputs.
G HIERARCHICAL CONCEPTS AND RELATIONS
4
T
Deft
Geftben:
k.6
Ceritema:
Figure 8: Illustration of relation clustering in the three datasets.
We give more insights on how relation clustering 2-ary predicates work here. In Figure 8 we show examples from the three datasets, we query a pair red and blue parts in the learned 2-ary predicate dictionary by comparing the distance of their representation obtained by the mapper and query layer and the 2-ary predicate prototypes, which gives a confidence distribution over each predicate, or a relative distance from them. One can also absolute distance with threshold to obtain better
21

---

Published as a conference paper at ICLR 2024
but-of-Diistrbuttio (OOD) detection ability for not only relation prediction but also other dictionaries and also better robustness Yang et al. 2018), for simplicity, we keep a relative distance our work.
On the left side of the figure, visualized the distribution in two LineWorld samples, where more common pattern "T" is close to predicate 3, while another overlapping pattern that is not allowed in the dataset is remote to all prototypes. On the right side, provide the confidence of the predicate with the largest confidence for each sample; we can see that the more common sample shows higher confidence than more random one which is hard be concluded as any categories.
While the 1-ary predicates learn visual primitives, the 2-ary predicates implicitly learn common combinations, and the higher-ary and order predicates can be our method can implicitly learn the hierarchy concepts (Lake et al. 2015)).
learning subparts. It shows that
H SYMBOL GROUNDING
P(Parallc))-e.uk --8.6
P(lerticaiEdg) -8.78 -8.78
P P(VerticalSep:)=B.cc
P(verticaliiccee.s
P(Yaralle))a0.66
Figure 9: Illustration of the symbol ground of an LW-G sample. Empty players are omitted.
Here, show how symbol grounding works that grounds image to set of predefined predicates. Figure shows an example of grounding an LW-G sample to the predicates defined by the baby ARC engine. The model first decomposes the image into parts, which are the lines different positions with different lengths, as I-ary predicates. Then predict the relationships between the pairs of the parts by the relation predictor which is implemented as 2-ary predicate prototypes.
This gives a complete graph Gs with I-ary concepts as nodes and 2-ary relationships as edges. Suppose the ground-truth graph is Ggt- Since the number of players is assumed to be larger than the number of ground-truth concepts, the model actually gives complete graph with outputs from all players as nodes Gp, and we require Ggz to be sub-graph of Gp and the nodes that are included in Gp but not Ggt to be empty. In training time, we extract the best matching subgraph of G to Ggt as Gy and compute the loss. In inference time, the prediction the complete graph the non-empty nodes. To extend our method to non-complete graph, may simply introduce threshold to relation prediction, or other OOD methods including using the absolute distance prototype classifier discussed in G simply introduce category for empty relation.
I ANALYSIS OF CLUSTERING INFORMATION GAIN
To calculate the CIG a model, we first decompose each sample x in the test set into parts स. resulting in the set X of all parts. We then generate set of parts X rand randomly sampling parts from samples in the test set, using a mask with normal distribution in each position. We then compute the MCE of both sets obtain MCE Emddel and MCErand: which allows calculate the CIG. To do this, we reduce the dimension of the parts and then run K-Means clustering.
The influence of dimension reduction faimess can be seen by comparing three typical dimension- reduction techniques: PCA, comparison of the three methods. The "Reference" in Line World is composed of common elements
(AE), and a are-trained CNN, VGG19. Table 5 provides a
22

---

Published as a conference paper at ICLR 2024
such lines and two vertical lines in the mid edge (i.e. "L" and "T" shapes) with different lengths, colors, positions, and directions; in OmniGlot, the "Reference" is the set of the ground-truth strokes. To test the stability of AE-based CIG, we trained AE ten times in each dataset and calculated the average results and variance. The results show that the AE-based CIG stable with a variance of approximately 2.85% across different runs.
Line World
OmniGlot
AE
VGG19
PCA
AE
VGG19
PCA
Reference
64.6 (+:3.5%%)
57.1
16.0
75.8 (+2.49%)
67.2
28.5
DFF
33.1 (#2.23%)
29.3
8.6
36.9 (#3.08%)
32.2
11.8
SCOPS
35.6 (+3.01%)
32.3
8.8
38.6 (+2.63%)
35.4
12.0
UPD
36.3 (+2.94%)
32.7
9.4
42.8 (+2.45%)
37.8
13.2
Ours
58.0 (+2279%)
51.5
13.0
68.5
(+3.32%)
60.6
20.3
Table 5: Experiment with different dimension reduction methods. On our method, DFF (Collins et al.
2018 SCOPS (Hung al. 2019), and UPD (Choudhury al. 2021
We utilized AE as the dimension reduction technique in our experiments because PCA was not satisfactory. Since the pre-trate tre-trained VGG can only be used for image data and AE yields similar outcome to VGG, we sought method that could provide a reasonable differentiation while being able to be applied to all types of data. We train shared AE when comparing different methods. For each dataset, we train an AE on that dataset and then use it to calculate and compare the CIG of different methods trained ShapeNet5, we set K to 32, and for World, we select 10, which is the number of components that we consider suitable for constructing the dataset.
the same dataset. We run K-Means with fixed K. For OmniGlot and
J ANALYSIS OF SHAPE SCORE
Irput
Inputt non-contiruous
M-spliie
0.493
Score 6.620
Score 0.650
Scons $39
Smooth 565
Smooth 750
Smooth 744
Smooth 904
Smooth 0.947 continucus & a solie
Search a 341 continnuus 8.577 So.id
conrinucus we
cancinuous
cantinuuus 3.400
Lontiruous 9.572
Irput fon-soiid
Conoes
B-Spline
Inout Gazs
Gorvex Hill
REF
B-Sp1inn Szome
Soore ass secech
Smooth 0.8GG
Secoth 0.ass
Seaoth 0.947
Conciruoue 5.0 Selti
continsoue 0-492
continupus 0-092
Solld 2. 1999
$o1id 2.000
so3ad 1.990
Figure 10: Visualization of the shape score under different cases and different smooth methods, the green lines are the smoothed shape.
We compare three ways to smooth the contour, minimal convex hull, -oonel-- Peucker (RDP) algorithm, and B B-spline interpolation in World and OmniGlot dataset, the result is listed in Table 6, where the three methods are marked as "Hull", "RDP" and "Spline", respectively, "Reference" is the same Table "Random" is obtained with the random sampling method to calculate the CIG.
23

---

Published as a conference paper at ICLR 2024
LineWorld
OmniGlot
Hull
RDP
Spline
Hull
RDP
Spline
Reference
89.7
99.5
97.9
80.0
86.1
82.9
Random
19.1
14.2
17.4
13.1
14.2
16.3
DFF
35.8
38.3
40.5
29.4
33.3
31.2
SCOPS
42.3
42.4
47.7
33.1
38.9
35.4
UPD
41.9
42.8
46.1
32.8
37.4
35.0
Ours
79.6
82.6
86.8
70.5
77.6
76.6
Table 6: Experiment with different contour smoothing methods. On our method, DFF (Collins et al. 2018), SCOPS (Hung al. 2019), and UPD (Choudhury al. 2021)
In our experiments, we apply RDP as the default smooth method, since the convex hull is too strict and prefers straight round shapes, B-Spline and RDP give similar results, but RDP gives slightly better differentiation. We further visualize the scores under different cases and different smooth we can see that RDP gives a smoother shape that fits the original shape better.
methods Figure 10
K EXAMPLES OF HUMAN EVALUATION CRITERIA
As shown the right. In each image, different strokes are marked with different colors, the same color means one stroke. There are four options for human evaluators to choose from, as follows, depending on whether they can redraw the character with given strokes.
Non-stroke
X M U
4
Unnatural
Non-stroke: They are not strokes at all; it is impossible draw the character with them.
Acceptable
Unnatural: Can draw with these strokes, but unnatural or uncomfortable
4 P
P
Acceptable: The strokes are not ideal enough, but not that unnatural.
Good
The strokes are close to those used by humans.
Good:
FTN
TINEE
Detailed instructions can be found in the Sup- plementary Material.
L LIMITATIONS AND BROADER IMPACTS
We discuss the limitations of the TDL framework in Section L.I and its broader impacts, as well as future directions in Section
L.1 LIMITATIONS
Data Insufficiency. TDL identifies compositional and reusable predicates through multi-ary clus- tering on the decomposed parts. Therefore, sufficient samples are required to form clusters, and TDL will not work when not provided with enough samples. For real-life data, more samples are needed to form robust clusters considering the noise. In contrast, humans are able to find compositionality in a few-shot manner and have high tolerance to the noise, by extrapolating the prior knowledge. This mystery that TDL. has yet uncover.
24

---

Published as a conference paper at ICLR 2024
Non-linear Composition. We assumee that the input z is linearly composed of parts. This simplifies the computation of the reconstruction error, as opposed to non-linear cases, where the reconstructed input needs to be obtained through a composition function that takes the set of compositions {x;), Nr input and produces the reconstructed input ã. It also makes it easier to represent combinations of compositions, since they can be simply added together. However, this may limit the representation power of the model, as viewing an image as a linear combination of parts may not capture the true generation process of the input. For example, in the real world, an image is the projection of a 3D world, so the parts should be 3D, and should be reconstructed like rendering process. Additionally, the linear assumption may not hold for other domains that also contain compositionality. such as language, audio, trajectories, etc.
Commonsense and Reasoning. Some predicates can only be discovered or grounded when given certain context. This context can be the environment, the cultural context, or common sense. Humans also infer the missing context deduce additional information from the input. For instance, when presented with a picture of a cat and an elephant, we can use our common sense of their sizes and the knowledge of perspective to compare their distance picture. This process requires a common sense of the size of the elephant and cat and reasoning ability using knowledge of perspective. However, as a representation framework, such predicates cannot be expected from TDL, as it has no common sense or a capacity to reason.
based on the size shown in the
L.2 BROADER IMPACTS AND FUTURE WORK
General Transitional Representation. The TDL can be extended beyond vision, as composi- tionality is present in many areas, not just vision. Vision is an intuitive case for to gain better understanding of compositionality in sigh-dimensional data. The TDL looks for compositional and reusable elements or combinations as predicates from the data by treating the sample as a bag of words and the dataset as a corpus. For instance, in robotics, a trajectory is composed of reusable actions, and certain combinations of actions known as skills. A decomposition model be trained to suggest potential decompositions of trajectories into actions using current predicate dictionaries, and then refined through clustering the decomposed actions and combinations. By defining the decomposition models, the TDL can be used to learn the transitional representation in different domains. Another potential future direction is learn general cross-domain transitional representation by clustering the embeddings of multi multi-modal data from different decomposition models.
Neural-Symbolic Pre-training. As an unsupervised representation learning framework, it is promising for TDL to scale up for large-scale pertaining. The foundation models that learn neural- symbolic transitional representations can provide better interpretability due to the embedded structural information well as the prototype predicate dictionaries. Furthermore, the learned representation is a pre-digging of the compositional information of the data in the pre-training, which can be reused in unseen tasks, and thus theoretically perform better for downstream applications.
Hidden Logical Rules. With the logical sentences of entities and relations grounded on the predicates learned by TDL., the model should be able reason when the rules are provided. Therefore, significant future work is learning hidden logical rules and reasoning with them in unsupervised manner. The domain related to unsupervised learning of rules is association rule learning (Agrawal al. 1993). which discovers rules such as X Y (where X and Y are sets of items) from the dataset. In the context of TDL, the items can be predicate prototypes. Instead of the joint probability P(X, Y). conditional probability P(Y X) can be used to model such rules in the likelihood computation in Equation
M QUALITATIVE COMPARISONS ON OMNIGLOT
We randomly selected 260 characters from the OmniGlot test set for comparison and qualitative analysis between model and the baseline models. Our model, which uses dictionary learning paradigm, can learn concepts such as lines and curves that are similar to human strokes. Each sample colored differently; however, the colors may blend together if pixel associated with different parts with varying levels of confidence. The less color mixing, the higher the confidence.
25

---

Published as a conference paper at ICLR 2024
M.1
OURS
1
1
26

---

Published as a conference paper at ICLR 2024
M.2
DFF
2 
STPK
22
55
27

---

Published as a conference paper at ICLR 2024
M.3
SCOPS
J 2
U
D
5
72
ف
28

---

Published as a conference paper at ICLR 2024
M.4
UPD
S
2 
29


Reviews:
[{"review_id": "WbOKeGwBIP", "reviewer": "Reviewer_6UkK", "summary": "This paper looks at a way to merge symbolic and DNN representations. The authors propose a transitional representation that contains high-fidelity details of the input and and also provides structural information about the semantics of the input. An Expectation-Maximization loop is used to optimize the parameters, where the Expectation step is used to optimize the hidden dictionary of parts, and maximize the overall likelihood of the dataset. To control the arity, techniques such as online clustering and random sampling are used. The authors conduct unsupervised segmentation on three abstract compositional visual object datasets and show superior accuracy compared to unsupervised clustering baselines.", "questions": "1. Are the predicates shared among different classes? Do predicates always correspond to semantic attributes? It would help to visualize the learnt 1-any and the 2-ary predicates. \nHow does the method compare to other neuro-symbolic baselines? The current set of baselines are essentially unsupervised clustering methods.\n2. \n2. Please provide a clear set of ablation studies which show the benefits drawn from each component. How can the system be simplified without affecting the overall accuracy.\n3. It would be good to have a limitations section that discusses when this method wouldn't work. How do predicates such as left_of and larger (examples from the paper) operate in case of multi-view settings, where these terms become ambiguous.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "Neuro-symbolic reasoning is a timely topic for research, and joint optimization of reconstruction and predicate logic appears to be an interesting idea. The method utilizes a dictionary of entities, and 1-any and 2-ary predicates as a neck to train the semantic distance during reconstruction. The works incorporates several interesting ideas such as Expectation Maximization, game-theoretic loss function and online prototype clustering to make the system work.", "weaknesses": "- The paper is a hard to read and the language is confusing. Technical concepts such as \"hidden dictionaries of symbolic knowledge\" are introduced early on without much explanation. \n- Experiments are limited to tiny, mostly binary datasets such as \"ShapeNet5\", which is basically a subset of 5 categories from the ShapeNet dataset. It is not clear if the methods would generalize to noisy real-world data, such as training using noisy, incomplete instances where the parts are not all visible. \n- Although the paper provides a reasonably well-curated list of neuro-symbolic approaches, the evaluations do not compare against any of the recent approaches. Instead the comparison is against clustering baselines.\n- The paper reads as a mishmash of several different ideas that are used together, but not integrated coherently. Therefore having a ablation studies to show the value of each module would be crucial. However, the evaluations do not provide a clear understanding of the contribution of each component to the overall methodology. \n- Ultimately, the task of reconstructing and explaining shapes simultaneously might be quite ambiguous as depicted in figure 4, and might not generalize to natural datasets, These aspects are not addressed in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C808qWBKRB", "reviewer": "Reviewer_jEsx", "summary": "The paper targets the reconstruction of an input signal $x$ (evaluated with images and point clouds) through a combination of parts in a learning framework. The solution is formulated as an unsupervised dictionary learning problem and solved through EM. The method is evaluated and compared on three datasets including 2D non-overlapping lines, 2D handwritten characters, and 3D shapes.", "questions": "Though Sec. 1&2 are well-demonstrated with clear motivations, the unsatisfied presentation of Sec. 3&4 makes the formulation and solution hard to follow. The authors are also encouraged to provide qualitative comparison results on Line World and ShapeNet5.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "+ The motivation and the background of the paper are well demonstrated and insightful in Sec. 1&2. The significance of the paper is clear and the arguments are insightful.", "weaknesses": "The major weakness of the paper is the bad presentation of Sec. 3&4 that greatly hinders the readers from understanding the paper.\n- The annotation in Sec. 3 is in quite a mess. Scalar value, vector, set, and matrix are not in consistent forms, and multiple critical variables lack clear definition/explanation:\n1) what is 'a' stands for in \"such as Cat(a), Tree(a), Person(a)\"? \n2) What is the relationship between $x_i$ and $x$? Seemingly the pieces of $x_i$ are determined by the masks and are directly combined into a whole instead of the linear addition.\n3) What is the relationship between $r_i$ above Eq. 1, $R_i$ in Eq. 1, and $r_j^i$ below Eq. 1?\n4) How can $theta$ be optimized in Eq. 1 if it does not appear in the two terms? The definition of the decoder g(·) is not consistent. Does it take $theta$ as a condition or not? Seemingly Eq. 2 is the appropriate form.\n5) The definitions of two crical terms $E_{\\tilde{D}}$ and $d_S$ are unclear.\n6) How is the dictionary $\\tilde{D}$ obtained given the argument \"As we have meaningful $\\tilde{D}$\"?\n7) It seems that the only variable to be optimized is the hidden dictionary $\\theta$. What about the models of $f(x;\\theta)$, $\\hat{g}(r_i;\\theta)$, $g_{\\theta}(R^i)$, and $g_{\\tilde{D}(R)}$?\n\n- The illustration of Fig. 2 does not clearly demonstrate the formulation in Sec. 3 and the solution in Sec. 4:\n1) $f, R, r_i, g, x_i, m_i$ are not clearly labeled in the figure.\n2)Where is the $N_P$ copies of the model in the figure? \n3) What does each patch stands for and what are the relation between the patches and the aforementioned terms in Sec. 3?\n4) Why is there a \"GT loss\" in an unsupervised learning pipeline?\n5) Where is the \"Decomposition Loss\" mentioned in Fig. 2?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JqUpmdVUgO", "reviewer": "Reviewer_KjjP", "summary": "This paper explores unsupervised part segmentation using a neural symbolic approach.  The authors propose Transitional Dictionary Learning for symbolic feature representations for representing the feature embedding as structural information.  This is done via a set of ‘players’ estimating the visual parts which are combined together for the reconstruction and clustering losses for self-supervised learning of the features.  In addition, a game-theoretic decomposition loss prevents one player from reconstruction everything or overlapping with other players.", "questions": "For Figure 3, could you compare a more conventional approach to compare against to see if this approach is causing it to be separated verse just from the data?  \n\nHave you tried this on more complex data 2D images?\n\nCan you elaborate on exactly what the human criteria were that they were evaluating?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and easy to understand.  There are good explanations for each step of the approach.  The \"Transitional Representation\" section does a really good job of approaching the symbolic and neural representations.  \n\n\nThe method is topical and will be of interest to the ICLR community and the method seems to be novel for how to produce a dictionary of neuro-symbolic part segmentation.  \n\nI really like the overarching goal for self-supervised part segmentation and the method seems to attack the problem directly.  The neural symbolic approach to ML has been of interest for a while and part segmentation is a good problem to apply it towards.", "weaknesses": "The biggest disappointment was not doing this on real visual data rather than on LineWorld data.  This is still useful with just LineWorld but showing on realworld data would be much more impressive.  \n\nRunning human evaluations requires an IRB or something similar not mentioned here.   This needs to be stated (anonymously) that you did actually go through someone to ensure the human experiments were done properly.  \n\nFor the “Compositional Representation” related work, please add references to older approaches such as Bag of Words such as:\nL. Fei-Fei and P. Perona, \"A Bayesian hierarchical model for learning natural scene categories,\" 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), San Diego, CA, USA, 2005, pp. 524-531 vol. 2, doi: 10.1109/CVPR.2005.16.\n\nCsurka, Gabriella, Christopher Dance, Lixin Fan, Jutta Willamowski, and Cédric Bray. \"Visual categorization with bags of keypoints.\" In Workshop on statistical learning in computer vision, ECCV, vol. 1, no. 1-22, pp. 1-2. 2004.\n\nThe citations needs to reference the actual venue such as this one should not just refer to Open Review (be wary of using automated citations):\nYann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62, 2022.\n\nFormular 1 -> Equation 1", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M3GWDgpyhl", "reviewer": "Reviewer_mxpA", "summary": "The paper introduces Transitional Dictionary Learning – a framework for implicitly learning symbolic knowledge, such as visual parts and relations, through input reconstruction using parts and implicit relations. This is done by employing a game-theoretic diffusion model for input decomposition, leveraging dictionaries learned by the Expectation Maximization (EM) algorithm. Experimental results demonstrate the proposed approach’s efficacy through evaluation in discovering compositional patterns compared to SOTA methods, depicting human alignment with the predictions as well.", "questions": "Apart from the points mentioned in the Weaknesses section, the paper could benefit from a broader discussion of its potential applications and impact, which would be valuable for the research community. Additionally, a more detailed analysis of the computational resources and time required would be helpful for readers seeking to implement the proposed methodology.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper provides a convincing motivation for the proposed methodology. It offers crucial insights into transitional representations, clustering information gain, and the reinforcement learning approach employed to optimize the objective. Overall, the paper exhibits a well-written supported by experimental evidence and a well-formulated mathematical framework. Figure 2, along with Section 4, elucidates the proposed approach and its crucial implementation details for the reader. I believe that this methodology holds significant promise for the research community, particularly in the midst of the surge of VLMs, where interpretable representations can not only serve as effective starting points or initializations, but also provide disentangled inputs for VLMs/LLMs to engage in high-level reasoning. The transfer learning experiments outlined in Table 2 provide strong evidence of the approach's utility beyond the confines of its training domain.", "weaknesses": "While the conducted experiments offer valuable insights into the effectiveness of the proposed approach, I would like to encourage the authors to extend their testing to more challenging real-world datasets. This expansion could further underscore the practical utility of the approach. Specifically, incorporating diverse categories of 3D objects from sources like ShapeNet, integrating written language datasets such as EMNIST, and including datasets featuring objects relevant to manipulation tasks would be valuable additions to the paper. Demonstrating the application of the proposed approach in contexts like robot manipulation or affordance prediction would provide tangible benefits for readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)