Compositional Law Parsing with Latent Random Functions
Authors: Fan Shi
Bin Li
Xiangyang Xue
Venue: ICLR 2023 poster (2023)

Abstract: Human cognition has compositionality. We understand a scene by decomposing the scene into different concepts (e.g., shape and position of an object) and learning the respective laws of these concepts, which may be either natural (e.g., laws of motion) or man-made (e.g., laws of a game). The automatic parsing of these laws indicates the model's ability to understand the scene, which makes law parsing play a central role in many visual tasks. This paper proposes a deep latent variable model for Compositional LAw Parsing (CLAP), which achieves the human-like compositionality ability through an encoding-decoding architecture to represent concepts in the scene as latent variables. CLAP employs concept-specific latent random functions instantiated with Neural Processes to capture the law of concepts. Our experimental results demonstrate that CLAP outperforms the baseline methods in multiple visual tasks such as intuitive physics, abstract visual reasoning, and scene representation. The law manipulation experiments illustrate CLAP's interpretability by modifying specific latent random functions on samples. For example, CLAP learns the laws of position-changing and appearance constancy from the moving balls in a scene, making it possible to exchange laws between samples or compose existing laws into novel laws.

Published as a conference paper at ICLR 2023
COMPOSITIONAL LAW PARSING WITH LATENT RANDOM FUNCTIONS
Fan Shi Bin Li Xiangyang Xue
Shanghai Key Laboratory of Intelligent Information Processing School of Computer Science, Fudan University fis 220m. fudan. edu. en {libin, xyxue ) @uddan.edu.cm
ABSTRACT
Human cognition has compositionality. We understand a scene by decompos- ing the scene into different concepts (e.g., shape and position of an object) and learning the respective laws of these concepts, which may be either natural (e.g.. laws of motion) or man-made (e.g., laws of game). The automatic parsing of these laws indicates the model's ability to understand the scene, which makes law parsing play central role in many visual tasks. This paper proposes deep la- tent variable model for Compositional LAw Parsing (CLAP), which achieves the human-like compositionality ability through an decoodng architecture to represent concepts in the scene as latent variables. CLAP employs concept- specific latent random functions instantiated with Neural Processes capture the law of concepts. Our experimental results demonstrate that CLAP outperforms the baseline methods in multiple visual tasks such as intuitive physics, abstract visual reasoning, and scene representation. The law manipulation experiments il- lustrate CLAP's interpretability by modifying specific latent random functions on samples. For example, CLAP learns the laws of position-changing and appear- ance constancy from the moving balls scene, making it possible exchange laws between samples or compose existing laws into novel laws.
INTRODUCTION
Compositionality is an important feature of human cognition (Lake et al., 2017). Humans can de- compose scene into individual concepts learn the respective laws of these concepts, which can be either natural (e.g. laws of motion) or man-made (e.g. laws of game). When observing a scene of moving ball, one tends parse the changing patterns of its appearance and position separately: The appearance stays consistent over time, while the position changes according to the laws of mo- tion. By composing the laws of the ball's appearance and position, one can understand the changing pattern and predict the status of the moving ball.
Although compositionallty has inspired number of models visual understanding such repre- senting handwritten characters through hierarchical decomposition of characters (Lake et al. 2011 2013) and representing multi-object scene with object-centric representations (Eslami al. 2016
Kosiorek et al.) 201 8: Greff al. 2019 Locatello al. 2020). automatic parsing of laws in scene is still a great challenge. For example, to understand the rules abstract visual reasoning such as Raven's 's Progressive Matrices (RPMs) test, the comprehension of attribute setrifute-specific representations and the underlying relationships among them is crucial for a model to predict the missing images (Santoro al. 2018 "Steenbrugge et al.! 2018: Wu al., 2020). To understand the laws of motion in intuitive physics, a model needs to grasp the changing patterns of different attributes (e.g. appear- ance and position) of each object in 2017 Ye et al.] 2018). However, these methods usually employ neural networks to directly model changing patterns of the scene in black-box fashion, and can hardly abstract the laws of individual concepts in an explicit, interpretable and even manipulatable way.
to predict the future (Agrawal et al.) 20166] Kubricht et al.)
A possible solution enable the abovementioned ability for a model is exploiting a function to represent a law of concept in terms of the representation of the concept itself. To represent a law
"Corresoonding author

---

Published as a conference paper at ICLR 2023
that may depict arbitrary changing patterns, an expressive and flexible random function is required. The Gaussian Process (GP) is a classical family of random functions that achieves the diversity of
function space through different kernel functions (Williams & Rasmussen, 2006). Recently pro-
posed random functions (Garnelo et al.
20185
Eslami et al. 2018:
Kumar et al. 2018 Garnelo
et al. 2018a; [ingh et al.] 2019 Kim al.
2019 Louizos al. 2019 Lee et al. 2020 Foong al.)
2020 describe function spaces with the powerful nonlinear fitting ability of neural networks. These
random functions have been used to capture the changing patterns in scene, such as mapping times- tamps to frames to describe the physical law of moving objects in a video (Kumar et al., 2018 |Singh al., 2019] Fortuin et al., 2020). However, these applications of random functions take images as inputs and the captured laws account for all pixels instead of expected individual concepts.
In this paper, we propose a deep latent variable model for Compositional LAAw Parsing (CLAP) CLAP achieves the human-like compositionality ability through an encoding-decoding architecture (Hamrick et al., 2018) to represent concepts in the scene as latent variables, and further employ concept-specific random functions in the latent space to capture the law on each concept. By means of the plug-in of different random functions, CLAP gains generality and flexibility applicable to various law parsing tasks. We introduce CLAP-NP functions with recently proposed Neural Processes (Garnelo et al.) 2018b).
example that instantiates latent random
Our experimental results demonstrate that the proposed CLAP outperforms the compared bascline methods in multiple visual tasks including intuitive physics, abstract visual reasoning, and scene representation. In addition, the experiment exchanging latent random functions concept and the experiment on composing latent random functions of the given samples to generate new samples both well demonstrate the interpretability and even manipulability of the proposed method for compositional law parsing.
specific
2 RELATED WORK
Compositional scene representation models (Yuan et al. 2022a) can understand a scene through object-centric representations (Yuan et al.) 22019] a |021 Emami et al. 2021 Yuan et al. 2022b). Some models initialize and update object-centric represen-
Compusitional Scene Representation
tations through iterative computational processes like neural expectation maximization (Greff et al.,
2017). iterative amortized inference (Greff al.) 2019). and iterative cross-attention (Locatello et al.,
2020 Other models adopt non-iterative computational processes to parse iisentangged attributes for
objects latent variables (Eslami al., 2016) or extract representations from evenly divided regions in parallel (Lin et al. 2020b). Recently, many models focus on capturing layouts of scenes (Jiang & Ahn 22200 or learning consistent object-centric representations from videos (Kosiorek et al. 20188] Jiang et al. 2019: [in et al., 2020a). Unlike CLAAP, compositional scene representation models learn scene are composed but cannot explicitly represent underlying laws and understand how these laws constitute the changing pattern of the scene.
how objects
Random Functions The GP (Williams & Rasmussen 2006) is a classical family of random func- tions that regards the outputs of function as a random variable of multivariate Gaussian distribu- tion. To incorporate neural networks with random functions, some models encode functions through global representations (Wu et al. 2018 Eslami et al. 2018 Gordon et al. 2019). NP (Garnelo et al. 2018b) captures function stochasticity with a Gaussian distributed latent variable. According to the way of stochasticity modeling, one can construct random functions with different characteristics (Kim et al. 2019, Louizos et al.] 2019 Lee et al.) 2020 Foong et af.) 2020). And other models develop random functions by learning adaptive kernels (Tossou et al. 2019) (ooo)//atacchhlaa
al.
2020
or computing the integral of ODES or SDEs on latent states (Norcliffe et al., [222] Li et al. 2020
Hasan al. 2021). Random functions provide explicit representations for laws but cannot model the compositionality of laws.
3 PRELIMINARIES
A random function describes distribution over function space F. from which we sample function f mapping the input sequence (.N, to output sequence (yI,...,yy) The impor-
Coode available http:///aiiiooootaaaaareaaeeeeee
2

---

Published as a conference paper at ICLR 2023
For each eonoept E C: parse the random z predict concept of langel points 25 using the concept- specific random function
Encode contical images to concepts E E
tafic to target images
Appearance
Background
Rotation
Azimuths
63
315'
135
106
198'
271'
Figure I: Overview of CLAP: to predict target images, CLAP first encodes the given context images into representations of concepts, such object appearance, background and angle of object rotation; then parses concept-specific latent random functions and computes the representations of concepts in the target images; finally decodes these concepts compose the target images.
tant role of a random function is uncovering the underlying function over some given points and predicting function values at novel positions accordingly. In detail, given set of context points Ds {( { E J), we should find the most possible function that y. f(xx)) ) for these con- text points, then use f(æ;) to predict yt: for target points Dr = {(x,. yt E T). The abovemen- tioned prediction problem can be regarded as estimating conditional probability p(yr|ær, Ds).
Neural Process (NP) NP (Garnelo et al. 2018b captures function stochasticity with a Gaussian distributed latent variable g and lets plyr Fr, Ds) [ p(y) (g) Ds) In this case, p(yrlar Ds) consists of generative process p(g 9 ate) and an inference process ((g) Ds), which can be jointly optimized through variational inference. To estimate p(yr rr, Ds). NP extracts the feature of each context point in Ds and sum them up to acquire the mean μ and deviation o of g:
H, - - Ty (
Σ Ta (ys. 2.) SES
(1)
Then NP samples the global latent variable g from [(g] Dg). NP independently predict ye the target position æ, through the generative process p(y: [g xx) V(y:TTT(g of) Neural networks Ta T and Tn increase the model capacity of NP learn data-specific function spaces.
MODEL
Figure the overview of CLAP. The encoder converts context images individual concepts in the latent space. Then CLAP parses concept-specifii random functions on the concept representations of context images (context concepts) to predict the concept representations of target images (target concepts). Finally, a decoder maps the target concepts to target images. The concept-specific latent random functions achieve the compositional law parsing in CLAP. We will introduce the genera- tive process, variational inference, and parameter learning of CLAP At the end of this section, we propose CLAP-NP that instantiates the latent random function with the Neural Process (NP).
4.1 LATENT RANDOM FUNCTIONS
Given Dy and Dr. the objective of CLAP is maximizing poyr( lys, 2r). In the framework of con- ditional latent variable models, we introduce a variational posterior q(hlpp j, x) for the latent variable and approximate the log- likeliild with evidence lower bound (ELBO) (Sohn et al. 2015):
log (hly,x)
log p (UT Us. .) (2(h)).)))) log (yr (yr(h)] By(holy ae)
L.
p (h (hlys ys
The latent variable h includes the global latent random functions f and local information of images z = {z,, ) where 2,, the low-dimensional representation of y,,. Moreover, we decompose 2, into independent concepts {= le C) (e.g., an image of single-object scene may have concepts
3

---

Published as a conference paper at ICLR 2023
Lakent Ramdom Function
9
45
x
X
X
CLAP
(b) NP as the latent random function
Figure 2: The graphical model of CLAP where the generative process is indicated using black solid lines and the variational inference indicated using red dotted lines. Panel (a) shows the framework of CLAP, where the latent random function fe for the concept c C can be instantiated by arbitrary random functions; Panel (b) describes how to instantiate the latent random function with NP.
such object appearance, illumination, camera orientation, etc.) where C' refers to the name set of these concepts. Assuming that these concepts satisfy their respective changing patterns, we employ independent latent random functions to capture the law of each concept. Denote the latent random function on the concept as S the specific form of f depends on the way we model it. As Figure 2b shows, if adopting NP the latent random function, a Gaussian-distributed latent variable to control the randomness of f Within the graphical model of CLAP in Figure 2a the prior and posterior in C are factorized into
have {(x,)
(9"; ax)) where g is
p(hlys, x:) = II
p ( æs) II (= 25,
(s, II p (=
OEC
LET
SES
(3)
(yr II P (2))z)), ((hly,æ)) = II EET
((1922 x)) II (( II a(=ilu)) CET
CEC'
eES
Decomposing f into concept-specific laws [ E C} in the above process critical for CLAP's compositionality. The compositionality makes possible parse only several concept-specific laws rather than the entire law on z, which reduces the complexity of law parsing while increasing the interpretability of laws. The interpretability allows us manipulate laws, for example, exchanging the law on some concepts and composing the laws of existing samples.
4.2 PARAMETER LEARNING
Based on Equations
2 and we factorize the ELBO as (See Appendix
B.3 for the detailed derivation)
(2(1)1))
L =
E (1(ppmm)) (velz))
Σ
Eg (hi w.m
E
log p (2f) fc, 23. æs, xx))
ICT
CECLET
Reconstruction term Cr
Target regularizer C.
(4)
log 21>g yy) (z (2)
log ((19220,0 w)) p pff(22g. 28)
E Ey(h
Context rogularizer
Function regularizer C,
Reconstruction Term C, is the sum of log (2:), which is modeled with a decoder to recon- struct target images from the representation of concepts. CLAP maximizes L, connect the latent space with the image space, enabling the decoder to generate high-quality images.
L. consists of Kullback-Leible (KL) divergences between the posterior (<f()))) and the prior pf=īff, #gg, Vy æ;) of target concepts. The parameter of g(2 (y:) is directly computed using the encoder contain more accurate information about the target image while p(z)) z5, xy, 2x) is estimated from the context. The minimization L, fills the gap between the posterior and prior to ensure the accuracy of predictors.
Target Regularizor
Context Regularizor E. consists of KL divergences between the posterior q( y.) and the prior p(25ly ys) of context concepts. To avoid mismatch between the posterior and prior and reduce model
4

---

Published as a conference paper at ICLR 2023
parameters, the posterior and prior are parameterized with the same encoder. In this case, have 32) = q(= We) and further L, = So we remove Lg from Equation
5 x)) and the prior Tg). We use f as a measure of function consistency to ensure that model can obtain similar distributions for fe with any subset 23 =" and æs shares concept-specific function parsers to model =0, x) and p(f9 Fg). And we need to design the architecture of function parsers so that they can take different subsets input.
Function Regularizor Lj consists of KL divergences between the posterior q 2f
inputs. To this end, CLAP
Correct decomposition of concepts is essential for model performance. However, CLAP is trained without additional supervision to guide the learning of concepts. Although CLAP explicitly de- fines independent concepts in priors and posteriors, it is not sufficient to automatically decompose concepts for some complex data. To solve the problem, we can introduce extra inductive biases to promote concept decomposition by designing proper network architectures, e.g., using spatial trans- formation layers to decompose spatial and non-spatial concepts (Skafte Hauberg. 2019). We can also set hyperparameters to control the regularizers (Higgins al. 2017), or add total correlation (TC) regularizer RTG to help concept learning (Chen et al. 2018). Here we choose to add extra regularizers to the ELBO of CLAP, and finally, the optimization objective becomes
argmax C = argmax f, - 3cC, - B - RTe P-q
(5)
P-q
where Bt. 1.......rc BTC are hyperparameters that regulate the importance of regularizers in the training process. We compute L through Stochastic Gradient Variational Bayes (SGVB) estimator (Kingma & Welling 2013) and update parameters using gradient descent optimizers (Kingma & Ba 2015).
4.3 NP AS LATENT RANDOM FUNCTION
In this subsection, propose CLAP-NP example that instantiates the latent random function with NP. According to NP, we employ a latent variable g to control the randomness of the latent random function f for each concept in CLAP-NP.
Generative Process
CLAP-NP first encodes each context image ys to the concepts:
(m5, /c E C} Encoder(y.))
8 EE
(6)
2 N (US...,?'))
CE C,C,
To stabilize the learning of concepts, the encoder outputs only the mean of Gaussian distribution, with the standard deviation as a hyperparameter. Using the process in Equation [ for each con- cept, a concept-specific function parser aggregates and transforms the contextual information the mean f4 and standard deviation oe of g". As Figure 2b shows, the sencept-specifie target pre- dictor Tp takes g" NW,, )iag(or)) and x. inputs to predict the mean م of æt. leaving the standard deviation as a hyperparameter. To keep independence between concepts, CLAP-NP introduces identical but independent function parsers and target predictors. Once all of the concepts =f N Nus o² I) for e E C' are generated, we concatenate and decode them into target images:
Yt ~ N ()) Hy.e = Decoder (f2î E C}), TE
To control the noise sampled images, we introduce a hyperparameter o, as the standard deviation. As Figure 2a shows, rewrite the prior p(hlys, x) in CLAP-NP as
II nEC
p 22 2s) II (2 xx) II p (2 [P(2))))) IET
(8)
sES
Inference and Learning In the variational inference, other than that the randomness of fo is replaced by g", the posterior of CLAP-NP is the same as that in Equation
(hly x) = II <0
" x) II c(( yv)
II ((=fll))) TET
cEC
SES
In the first stage, we compute the means of both F5 and žy using the encoder CLAP. Because the encoder is shared in the prior and posterior, we obtain 25 through in of recalculating
5

---

Published as a conference paper at ICLR 2023
Table 1: MSEs on BoBa, CRPM, and MPI3D dataset. The training configurations of datasets are given in the brackets, and the test configurations are displayed in the table headers.
BoBa-1 by) 1/12~>4/12)
BoBa-2 (1) = 1/12-44/12)
CREM-DI (1) = 1/9~2/9)
Model
4=4/12
7= 6/12
1=4/12
9==6/12
=/9
7) = 3/9
NP
420.7 = 4.4
649.2 + 6.0
1169.6 + 174
1917.8
24.8
59.3 + 4.7
101.9 + 12.2
GP
1165.5 +: 78.4
2064.9
76.6
2062.6 1 137.1
3556.8 1: 132.8
263.2 1 19.0
425.3 1: 22.0
36.6
22,9
3104.5 40.9
CLAP-NP
70.7 1: 2.7
151.2 1: 11.1
912.0 1 52.4
18338
91.9
19.6 1. 0.7
49,5 10.7
CRPM-DC 1/9~229)
MPI3D 1/8mx488)
Model
2/9
1 = 3/9
1)
11 6/8
1 10/40
1| 2040
NP
192.6 13.1
131.5
GP
468.3 1: 40.6
764.5 1 39.0
268.9 1: 13.0
491.6 1: 12.9
279.3
8.2
650.4 1: 12.5
GQN
414.3 7.2
823.0 + 15.3
55.6 0.7
331.8
772.1
1148,9 3.2
CLAP-NP
119.5 17.2
264.5 1 27.9
58.8 2.3
153.2 6.3
105.5
0.6
213.9 1.0
BoBo
BoBo
CRPM DT
CLAP MP
CRPM6-C
MP13D
CLA-UP
Figure 3: SA scores BoBa, CRPM, and MPI3D dataset where the blue, orange, purple, and green lines denote scores of CLAP-NP, NP, GP, and GQN.
2, ~.~ (>1g)))) Then we compute the distribution parameters of g" through the shared concept- specific function parsers and the input 2 and With the above generative and inference processes, some subterms in the ELBO of CLAP-NP become
4= Σ Σ E Ey(h/p..)) log (<2f1y))
log 2( p(g1255 xs))
Σ EEC
(10)
p xx:)
nEC LE
EXPERIMENTS
To evaluate the model's ability of compositional law parsing on different types of data, we use three datasets in the experiments: (1) Bouncing Ball (abbreviated BoBa) dataset (Lin et al., 2020a) to validate the ability intuitive physics; (2) Continuous Raven's Progressive Matrix (CRPM) dataset (Shi al., (021)) to validate the ability of abstract visual reasoning; (3) MP13D dataset (Gondal et al. 2019) to validate the ability of scene representation. We adopt NP (Garnelo et al.) 2018b), GP with the deep kernel (Wilson al. 2016), and GQN (Eslami et 2018) baselines. We consider NP and GP as baselines because they are representative random functions that construct the function space in different ways, and CLAP- instantiates the latent random function with NP. GQN is adopted since it models the distribution of image-valued functions, which highly fits our experimental settings. NP not used model image-valued functions, so we add an encoder and decoder to NP's aggregator and predictor to help it handle nighddimensional images. For GP, a pretrained encoder and decoder to reduce the dimension of raw images. See Appendix [DI and D.2 for detailed introduction datasets, hyperparameters, and architectures of models.
For quantitative experiments, we adopt Mean Squared Error (MSE) and Selection Accuracy (SA) as evaluation metrics. Since non-compositiooall law parsing will increase the risk of producing unde- sired changes on constant concepts and further cause pixel deviations in predictions, we adopt MSE indicator of prediction accuracy. To compute the concept-level deviations between predictions

---

Published as a conference paper at ICLR 2023
CAN
Figure 4: Prediction results (in red boxes) on BoBa-2 with 1j 4/12 (left) and 1) 6/12 (right).
X
4
4
Paslatan 2
Profetiie
Prsternn
Pind
Paokiinn
Piid
Figure 5: Prediction results (in red boxes) of CRPM-DT with 7) 2/9 (top) and 17 3/9 (bottom). We provide two predictions of the same context to illustrate special case: when a row of images is removed from the given matrix, all predictions with the progressive law of size and constant law of color are correct. In this special case, only CLAP-NP captures such prediction uncertainty.
and ground truths, which we think can be an indicator of semantic error, we introduce SA as another metric to calculate the distance on the representation space. Since the scale of representation-levv distances is tightly according to the representation space of models, we turn to get SA by selecting target images from a set candidates. For (ær, y), we combine the target images yr with K targets randomly selected from other samples to establish candidate set. Then the candidates and the pre- diction ýr ~ p(yrlys x) of the models are encoded into representations to calculate concept-level distances. Finally, the candidate with the smallest distance is chosen, and SA is the selection accu- racy across all test samples. In the following experiments, we use r = /N/N to denote the training or test configuration where Ny of N images in the sample target images predict.
5.1 INTUITIVE PHYSICS
To evaluate the intuitive physics of models, which is the underlying knowledge for humans to un- derstand the evolution of the physical world (Kubricht et al. 2017), we use BoBa dataset where physical laws of collisions and motions are described as functions mapping timestamps frames. BoBa-1 and BoBa-2 are one-ball and two-ball instances of BoBa. When predicting target frames, models should learn both constant laws of appearance and physical laws of collisions. We set up two test configurations = 4/12 and 6/12 for BoBa. The quantitative results in Table [ and Fig- ure illustratee CLAP-NP's intuitive physics on both BoBa-l and BoBa-2. Figure 4 visualizes the predictions of BoBa-2 where NP only generate blurred images while GP and GQN can hardly understand the appearance constancy of balls. That is, balls in the predictions of GP have different colors, and GQN generates non-existent balls and ignores existing balls. Instead, CLAP-NP can separately capture the appearance constancy and collision laws, which benefits the law parsing in scenes with novel appearances and physical laws (Appendix E.7
5.2 ABSTRACT VISUAL REASONING
Abstract visual reasoning is manifestation of human's fluid intelligence (Cattell 1963). And RPM (Raven & Court 1998) is a famous nonverbal test widely used to estimate the model's abstract visual reasoning ability. RPM is 3 x image matrix, and the changing rule of the entire matrix consists of multiple attribute-specific subrules. In this experiment, we evaluate models on four instances of the CRPM dataset and two test configurations 7) = 2/9 and = 3/9. In Table and Figure 3 CLAP-NP achieves the best results with the metrics in both MSE and SA. Figure displays the predictions CRPM-DT. In the results, NP produces ambiguous-edgen outer triangles, GP predicts targets with incorrect laws, and GQN generates multiple similar images. It is worth stressing that the answer

---

Published as a conference paper at ICLR 2023
Traced 2
Louel Dnsemaatio Chaina
Labest To EICIA
Figure 6: The qualitative latent traversal results and quantitative variance declines on BoBa-2, CRPM-T, and CRPM-C. High variance declines indicate significant correlations between laws and LRFs, by which we can determine whether dimension encodes the law we want to edit.
an RPM is not unique when we remove the row from it. We display examples of this situation in the second row of Figure [] indicating that CLAP-NP can understand progressive and constant laws on different attributes rather than simple context-to-targe mappings.
5.3 SCENE REPRESENTATION
In scene representation, we use MPI3D dataset where a robotic arm in scenes manipulates the al- titude and azimuth of the object to produce images with different laws. The laws of scenes can be regarded as functions that project object altitudes or azimuths RGB images. In this task, the key to predicting correct target images is to determine which law the scene follows by observing given con- text images. We train models under 1/= = 1/8-~/8 and test them with more target or scene images (1) = 4/8, 6/8, 10/40, 20/40). The MSE and SA scores in Table and Figure Jauueeedemonstaaa that CLAP-NP outperforms NP in all configurations and GQN in 17- = 6/8, 10/40, 20/40. CLAP-NP has a slightly higher SA but lower MSE scores than GQN when 17 4/8. A possible reason is that the network of GQN has good inductive biases and more parameters to generate clearer images and achieve better MSE scores. However, such improvement in the pixel-level reconstruction has limited influence on the representation space, as well as the SA scores of GQN. GQN can hardly generalize the random functions novel situations with more images, leading to performance loss when 1) 6/8 10/40, 20/40. GP's MSE scores on MPI3D indicate that it is suitable for scenes with more context images (e.g., 77 = 10/40). With only sparse points, the prediction uncertainty of GP leads to significant decrease in performance. However, compositionality allows CLAP-NP to parse concept-speciff subrules, which improves the model's generalization ability in novel test configurations.
5.4 COMPOSITIONALITY OF LAWS
It is possible exactly manipulate a law if one knows which latent random functions (LRFs) encode it. We adopt two methods to find the correspondence between laws and LRFs. Figure 6 visualizes the changing patterns by perturbing and decoding concepts, enabling us understand the meaning of LRFs. Another way is to compute variance declines between laws and LRFs (Kim & Mnihj 2018). Assume that we have ground truth laws and a generator fixing the arbitrary law / E L. First, we generate a batch of samples without fixing any laws and parse LRFs for these samples to estimate the variance 7f of each concept-specine LRF. By fixing
produce sample batches by
the law 1, we generate some batches (B)) 6=1 and estimate the in-batch variance (uf e C} of
LRFs for By. Finally, the variance decline between the law and the concept is
Na
1
81.c
(11)
NB
Jo
In Figure 6 we display the variance declines on BoBa-2, CRPM-T, and CRPM-C, which will guide the manipulation of laws in the following experiments. And Figure visualizes the law manipulation process, which well illustrates the motivation of our model on compositional law parsing.
Latent Random Function Exchange The top of Figure 7 shows how we swap the laws of sam- ples with the aid of compositional law parsing. To exchange the law of appearance between samples, first refer to the variance declines in Figure For BoBa-2, the laws are represented with 6 LRFs
8

---

Published as a conference paper at ICLR 2023
Recomtrat Tane
Drodkt Saap
Composikond Law Pursnn
Law Low
Low
Low
Indange Lits bevon
Tetepeliate
veeplo
Collus
Compooitedd
Lev Parikg
Loe Teeiry
Low Deiikg
Lan Poing
Poodic
Figure 7: An illustration of law manipulation. At the top, we exchange the law of appearance by swapping the LRFs of two samples on the first and last dimensions. At the bottom, we compose laws from existing samples of CRPM-T and CRPM-C to generate novel samples.
where the first and last LRFs encode the law of appearance while the others indicate the motion of balls. Thus we infer the LRFs (global rule latent variables in CLAP-NP) of two samples and swap the LRFs of two samples on the first and last dimensions. Finally, we regenerate the edited samples using the generative process, and further, make predictions and interpolations them.
Latent Random Function Composition For samples of CRPM, we manipulate laws by combin- ing the LRFs from existing samples to generate samples with novel laws. Figure the results of law composition on CRPM-T. The first step is parsing LRFs for given samples. By querying the variance declines in Figure 6 we can find these LRFs respectively correspond to the laws of size, rotation, and color. Since we have figured out the relations between laws and LRFs, the next step is to pick laws from three different samples respectively and compose them into unseen combi- nation of laws. Finally, the composed laws are decoded to generate a sample with novel changing patterns. This process is similar to the way human designs RPMs (i.e., set up separate sub-laws for each attribute and combine them into complete PRM). From another perspective, law composition provides an option for us generate novel samples from existing samples (Lake et al. 2015).
CONCLUSION AND LIMITATIONS
Inspired by the compositionality in human cognition, we propose deep latent variable model for Compositional LAw Parsing (CLAP). CLAP decomposes high-dimensional images into indepen- dent visual concepts in the latent space and employs latent random functions to capture the concept- changing laws, by which CLAP achieves the compositionality of laws. To instantiate CLAP, we propose CLAP-NP that uses NPs as latent random functions. The experimental results demonstrate the benefits of our model in intuitive physics, abstract visual reasoning, and scene representation. Through the experiments on latent random function exchange and composition, we further qualita- tively evaluate the interpretability and manipulability of the laws learned by CLAP..
Limitations. (1) Complexity of datasets. Because compositional law parsing is an underexplored task, we first validate the effectiveness of CLAP on datasets with relatively clear and simple laws to avoid the influence of unknown confounders in complicated datasets. (2) Setting the number of LRFs. For scenes with multiple complex laws, we can empirically set an appropriate upper bound or directly put large upper bound on the number of LRFs. However, using too many LRFs increases model parameters, and the redundant concepts decrease CLAP's computational efficiency. See Appendix | for detailed limitations and future works.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China (No.62176060), STCSM projects (No.20511 1000400, No.2251 1105000), Shanghai Municipal Sci- ence and Technology Major Project (2021 (H2DZZX00)))) and the Program for Professor of Special Appointment (Eastern Scholar) at Shanghai Institutions of Higher Learning.
REFERENCES
Pulkit Agrawal, Ashvin V Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physies. Advances in neural information processing systems, 29, 2016.
Raymond B Cattell. Theory of fluid and crystallized intelligence: A critical experiment. Journal of educational psychology. 54(1):1, 1963.
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disen- tanglement in variational autoencoders. In Advances in Neural Information Processing Systems, pp. 2610-2620, 2018.
Patrick Emami, Pan He, Sanjay Ranka, and Anand Rangarajan. Efficient iterative amortized in- ference for learning symmetric and disentangled multi-object representations. In International Conference on Machine Learning, pp. 2970-2981. PMLR, 2021.
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. Advances Neural Information Processing Systems, 29, 2016.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, (2006669)::104-1222
2018.
Andrew Foong, Wessel Bruinsma, Jonathan Gordon, Yann Dubois, James Requeima, and Richard Turner. Meta-leaming stationary stochastic process prediction with convolutional neural pro- cesses. Advances in Neural Information Processing Systems, 888284-8295.. 2020.
Vincent Fortuin, Dmitry Baranchuk, Gunnar Rätsch, and Stephan Mandt. Gp-vae: Deep probabilis- tic time series imputation. In International conference on artificial intelligence and statistics, pp- 1651-1661, PMLR, 2020.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference on Machine Learning, pp. 1704-1713. PMLR, 2018a
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv:1807.01622, 2018b.
Muhammad Waleed Gondal, Manuel Wuthrich, Djordje Miladinovic, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation the real world: new disentanglement dataset. Advances in Neural Information Processing Systems, 32, 2019.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Turner. Meta- learning probabilistic inference for prediction. In International Conference on Learning Repre- sentations, 2019.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. Advances in Neural Information Processing Systems, 30, 2017.
Klaus Greff, Raphaël Lopez Kaufmann Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning, pp. 2424 2433.. PMLR, 2019.
10

---

Published as a conference paper at ICLR 2023
Jessica B Hamrick, Kelsey R Allen, Victor Bapst, Tina Zhu, Kevin R McKee, Joshua B Tenen- baum, and Peter W Battaglia. Relational inductive bias for physical construction in humans and machines. arXiv preprint arXiv:1806.0122 2018.
Hasan, João M Pereira, Sina Farsiu, and Vahid Tarokh. Identifying latent stochastic differential equations. IEEE Transactions Signal Processing, 70:89-104, 2021.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017.
Jindong Jiang and Sungjin Ahn. Generative neurosymbolic machines. Advances in Neural Informa- fion Processing Systems, 13:12522-12582, 2020.
Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. arXiv preprint arXiv:1910.02384,, 2019.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. arXiv preprint arXiv: 2018.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Gamelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In International Conference on Learning Representations, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. 13Xiv:1312.6114, 2013.
arXiv preprint
Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. Advances in Neural Information Processing Systems, 31, 2018.
James R Kubricht, Keith J Holyoak, and Hongjing Lu. Intuitive physics: Current research and controversies. Trends in cognitive sciences, 21(10):449-75, 2017.
Ananya Kumar, SM Eslami, Danilo J Rezende, Marta Gamelo,, Fabio Viola, Edward Lockhart, and Murray Shanahan. Consistent generative query networks. arXiv preprint arXiv: 1807.02033, 2018.
Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. In Proceedings of the annual meeting of the cognitive science society, volume 33, 2011.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266): 1332-1338. 2015.
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Juho Lee, Yoonho Lee, Jungtack Kim, Eunho Yang, Sung Hwang, and Yee Whye Teh. Boot- strapping neural processes. Advances in neural information processing systems, 33:6606-6615, 2020.
Xuechen Li, Kam Leonard Wong, Ricky TQ Chen, and David Duvenaud. Scalable gradients for stochastic differential equations. In International Conference on Artificial Intelligence and Statistics, pp. 3870-3884 PMLR, 2020.
Zhixuan Lin, Yi-Fu Wu, Skand Peri, Bofeng Fu, Jindong Jiang, and Sungjin Ahn. Improving gen- erative imagination in object-centric world models. In International Conference on Machine Learning, pp. 6140-6149, PMLR, 2020a.
11

---

Published as a conference paper at ICLR 2023
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang. and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. arXiv preprint vrXiv:201..02407 2020b.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. Advances in Neural Information Processing Systems, 33: 1525-11538, 2020.
Christos Louizos, Xiahan Shi, Klamer Schutte, and Max Welling. The functional neural process. Advances in Neural Information Processing Systems, 32, 2019.
Alexander Norcliffe, Cristian Bodnar, Ben Day, Jacob Moss, and Pietro Liò. Neural ode processes. In International Conference on Learning Representations, 2021.
Massimiliano Patacchiola, Jack Turner, Elliot J Crowley, Michael O'Boyle, and Amos J Storkey. Bayesian meta-leaming for the few-shot setting via deep kernels. Advances in Neural Information 2020.
Processing Systems, 33:16108-166
John C Raven and John Hugh Court. Raven's 's progressive matrices and vocabulary scales, volume 759. Oxford pyschologists Press Oxford, 1998.
Adam Santoro, Felix Hill, David Barrett, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In International Conference on Machine Learning, pp. 4477-4486, 2018.
Fan Shi, Bin Li, and Xiangyang Xue. Raven's s progressive matrices completion with latent gaussian process priors. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 9612-9620, 2021.
Gautam Singh, Jaesik Yoon, Youngsung Son, and Sungjin Ahn. Sequential neural processes. Ad- vances Neural Information Processing Systems, 32, 2019.
Nicki Skafte and Søren Hauberg. Explicit disentanglement of appearance and perspective in gener- ative models. Advances in Neural Information Processing Systems, 32, 2019.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28: 3483-3491, 2015.
Xander Steenbrugge, Sam Leroux, Tim Verbelen, and Bart Dhoedt. Improving generaliza- tion for abstract reasoning tasks using disentangled feature representations. arXiv preprint arXiv:1811.04784 2018
Prudencio Tossou, Basile Dura, Francois Laviolette, Mario Marchand, and Alexandre Lacoste. Adaptive deep kernel learning. arXiv preprint 10X.v:1905.12131. 2019.
Christopher K Williams and Carl Edward Rasmussen. Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA, 2006.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pp. 370-378. PMLR, 2016.
Tailin Wu, John Peurifoy, Isaac L Chuang, and Max Tegmark. Meta-learning autoencoders for few-shot prediction. arXiv preprini arXiv:1807.0900 1807.0991. 2018.
Yuhuai Wu, Honghua Dong, Roger Grosse, and Jimmy Ba. The scattering compositional
leamner: Discovering objects, attributes, relationships in analogical reasoning. arXiv preprint 22X.v22007.04212, 2020.
Tian Ye, Xiaolong Wang, James Davidson, and Abhinav Gupta. Interpretable intuitive physics model. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 87-102, 2018.
12

---

Published as a conference paper at ICLR 2023
Jinyang Yuan, Bin Li, and Xiangyang Xue. Generative modeling of infinite occluded objects for compositional scene representation. In International Conference on Machine Learning, pp. 7222- 7231. PMLR, 2019a.
Jinyang Yuan, Bin Li, and Xiangyang Xue. Spatial mixture models with learnable deep priors for perceptual grouping. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 9142 2019b.
Jinyang Yuan, Bin Li, and Xiangyang Xue. Knowledge-guidde object discovery with acquired deep impressions. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 10798-10806, 2021.
Jinyang Yuan, Tonglin Chen, Bin Li, and Xiangyang Xue. Compositional scene representation learning via reconstruction: A survey. arXiv preprint arXiv:2202..77135, 2022a.
Jinyang Yuan, Bin Li, and Xiangyang Xue. Unsupervised learning of compositional scene represen- tations from multiple unspecified viewpoints. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8971 8971-892 19999, 2022b.
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
In Supplementary Materials, we (1) provide the details of ELBO, then (2) introduce the datasets, model architecture, hyperparameters, and computational resources adopted in our experiments, and finally (3) provide additional experimental results. In particular, provide additional results of editing and manipulating latent random functions, which validate our motivation and contribution.
B DETAILS OF ELBO
B.I ELBO OF CONDITIONAL LATENT VARIABLE MODELS
@ (hly,, x)) log (yr Us. x) dh log p (yr Us. x)
(h, yT ys 2t)
q (hl x)) log
dh
p (h y, x)
(hh, yr ys
dh + J 9 (hlg, log
q qhlly. dh (12)
q (hl x)) log
q (hly,, x)
p p( x)
(h, yr ys x)
q (hl f, x)) log
dh
q (hly.
log q (hl x))
B llog (3r [h)] (b)l,.))
L.
p phlyys,
B.2 PRIOR AND POSTERIOR FACTORIZATION
hh (v))z)
RET
p(h ys.
II p (f 25z 271ys. cEC
II cEC
p (2 e 2SS us, ST) (f 225 as) (zglus)
II
p ( 0s
II
p (= 25. æs, 20,) II p (2]lus)
(13)
IET
q( x)
II qfS 25. 1)
CEC
Il cEC
a (119)2 20 ())( (
II q(f912= xz) II (<))lg.) (=f(l))) cEC
#ES
IET
B.3 ELBO OF CLAP
Eg(h y. log II (y)(2)))
IET
(14)
Ncec (g(5) x:) ITe ( Ihr q(2 vc))
- Eg(hly.e))
log ILEe (Thrr p (2f) æs n. p (fo12 2s)) Iles P (2(1)=))
14

---

Published as a conference paper at ICLR 2023
Ier g(
B (1) logp (ye [0ilze]]
Σ (hllet
log
I P (z+) f z 2ss xta)
1ET
OEC
log 10 ,2)
II o (=2 ys)
(1 w.)
log
p (ff x5z xs)
ITES (x:² yss
DEC
CEC
Eychly.a) log
g(z (3b))
Be(h)y..)) logg (y)lle)))))
p (=f150 z ass, (t:)
IET
SEC LET
Reconstruction for
Target regularizer
Be( log ((=glv,)
log 9(1921, p 25,
IR (1(
p (22 ys)
cEC #ES
GEC
Context regularizer C.,
Function regulatizer L.
B.3.1 RECONSTRUCTION TERM
L
2 h m.))
logp (v(l))) Σ E (2( log (3olz1)
T
EE
(15)
2
log (B)(Ee)
where &s ~ g( 13)))
AE
B.3.2 TARGET REGULARIZER
log ((=710)) p (2 faallff2200 25 ss xee)
4= = 4=EY
E (g(h)
IET
(16)
(=(()1)) p (zf) f 25. ss x)
ΣΣ CECTET
Bocaglorr) E (=Glus)))
Barre log
Because of the function consistency in samples, function posteriors off* 3 x) derived from arbi- trary sampled W" q(="ly) need to be consistent, So we let f condition on only the observations and y to simplify the posterior distribution, that is, we replace (()022,
, x)) with q(f (
g (v)
C, ~= Σ
E Egt=glnes
Eg(zplor))
B
log p (2 F,,,,,$.55 Cs, x.,)
cEC tE
(17)
Σ eEC EET
E Co (Ms)
Eg(F+/
(L(((2Z) (2flif 25. xg xx,)
In this way, we convert the computation of the log-likelihoods on g(=((u)) and p(=f15 = ass mx) to the KL divergences between them. Then a Monte Carlo estimator can be used to approximate C, where zG is sampled by z q(>šlys) (ys to compute the outer expectation Eg(zglvs) | and for inner expectation Eg(F")y.))]]] [+]. because qff"',, x) ff
q(f"12" ( ())g(=)))d22"" we are able first
sample 2G ~ ((>))1r)) and obtain f ~ ( æf, By means of f and øe sampled from the posterior, we have L. s (I.( fe, "5. ans, xx)).
B.3.3 CONTEXT REGULARIZER
log (<f(ly)) p y.)
C. c. Bn(he
ΣΣΣ
E ( [oog]]
(18)
OEC SES
+ES
15

---

Published as a conference paper at ICLR 2023
B.3.4 FUNCTION REGULARIZER
log (f
L = Σ
E By(hhly.r)
p (8ol z5z, as))
cEC
log (f x) p ( as) KL (Lf ( (f9 xs))
Σ cEC
B By(=$lus (s)
B [(( (
(19)
Σ E (z=l)l
<EC
To estimate C in the same way as the target regularizer, 2" is sampled from g(=1lg) and we have y = (L x) (f xs)).
B.4 TOTAL CORRELATION
Let {y denote the set of all sample images in dataset, and žĄ. ~ g(zx: yv are latent representations of concepts for the kth image. To apply the Total Correlation (TC) (Chen et al.)
2018 in CLAP, decompose the aggregated posterior @(z) Σ (2 (((k)))))))))))))))))))))))))) according to the concepts, that is
Rrc KL. ((2)
[I( a(=9)
II
Rg(()) [og
Σ Ex(()) [lg ((9)( oEC
(20)
CEC
We adopt Minibatch Weighted Sampling (Chen al., 2018 to approximate Ric on the minibatch S I of the dataset and the corresponding representations of concepts { { }M Jm-
L. (ym mmml
By(x) logg [[=)]
log
q (2) log(KM))
M
(21)
E Em(=) (og)(>²))
log Σ (2fly,) log(KM)
M
C PRELIMINARIES
Generative Query Network (GQN) GQN (Eslami et al. 2018) regards the mappings from cam- era poses to scene images as random functions. GQN adopts deterministic neural scene repre- sentations r and latent variables z capture the configuration and stochasticity of scenes in the conditional probability p(r @rr, yc, xc) = p(yr. 2 ÆT r)dz. The representation network Ta extracts the representation
each context point and summarizes them as the global scene represen- tation = >nEG (ye, Z.). The generation network T, predicts the mean of target scene images via Tr(r,, z, a.). and the target scene images are sampled from A o²I).
Gaussian Process (GP) GP (we only discuss noise-free GP here) models the probability distribu- tion p(y|2) (z) the multivariate Gaussian ~ (O K) where Kij is computed through the kernel function k(x,, æz). In this case, the probability p(yr| yc, &c) ) a multivariate Gaussian, and the parameters have closed-form solutions. Kernel functions
the key to constructing different types of random functions. The models can use a basic kernel like the RBF kernel, combine dif- ferent kernlss develop complex kernels, or learn the function space for different data adaptively through neural networks (Wilson et al. 2016).
D DATASETS AND EXPERIMENTAL SETUP
D. DETAILS OF DATASETS
In this paper, three types of datasets adopted: BoBa, CRPM. and MPI3D. Figure B displays two samples for each instance datasets, and Table describes the train, validation, and test configura- tions of all instances.
16

---

Published as a conference paper at ICLR 2023
(a) BoBa-1
BoBa-2
A
(c) CRPM-T
(d) CRPM-DT
(e) CRPM-C
(i) CRPM-DC
(g) MPI3D
Figure 8: Examples from different instances of datasets. BoBa includes instances (a) BoBa-1 and (b) BoBa-2; CRPM includes four instances (c) CRPM-T, (d) CRPM-DT, (c) CRPM-C, and (f) CRPM-DC: (g) we only use one instance of MPI3D.
17

---

Published as a conference paper at ICLR 2023
Table 2: The detail of BoBa, CRPM, and MPI3D. Row I: dataset names. Row 2: splits of datasets where Test-k denotes that there are k target images in one sample. Row 3: the number of samples in each split. Row 4: the number of images in each sample. Row the number of target images in sample, Row 6: the size of images denoted as Channel x Height x Width.
Dataset
BoBa-I
BoBa-2
Split
Train
Valid
Test-4
Test-6
Train
Valid
Test-4
Test-6
Samples
10000
1000
2000
2000
10000
1000
|
2000
2000
Images
12
12
Targets
1 4
1
6
~
4 |
Image Size
x 64
x 64 64
Dataset
CRPM-T
CRPM-C
Split
Train
Valid
Test-2
Test-3
Train
Valid
Test-2
Test-3
Samples
10000
1000
2000
2000
10000
1000
2000
2000
Images
Targets
1 2
1~2
2
3
1~2
~~
2
Image Size
1 x 64 x 64
1 x 64 64
Dataset
CRPM-DT
CRPM-DC
Split
Train
Valid
Test-2
Test-3
Train
Valid
Test-2
Test-3
Samples
10000
1000
2000
2000
10000
1000
2000
2000
Images
Targets
1~ 2
1
2
3
1~2 2
1 |
Image Size
1 x 64 x 64
1 1 64
64
Dataset
MPI3D
Split
Train
Valid
Test-4
Test-6
Test-10
Test-20
Samples
16127
2305
4608
4608
4608
4608
Images
8
40
Targets
1 1~4
~ 4
4
10
20
Image Size
3 64 x 64
Table 3: A detailed description of æn on datasets.
Dataset
Deseription
CRPM
2D index of the image in matrix timestamps of the frame
BoBa
MPI3D
azimuths or altitudes of the center object
Bouncing Ball (BoBa) BoBa contains videos of bouncing balls. Depending on the number of balls in the scene, the dataset provides two instances BoBa-l and BoBa-2. In the videos of Figure 8a and [bb] the motion of balls follows the law of physical collisions; and the appearance of balls (color, size, amount, etc.) is constant over time. The models need to capture the probability space on functions f R + 33x64464 that map timestamps to video frames. Referring to Table we provide 10,000 videos of bouncing balls for training, 1,000 for validation and hyperparameter selection, and 2,000 to test the intuitive physics of the models. In the training and validation phase, we randomly select 1 to 4 frames from video frames the target images to predict; in the testing phase, provide two configurations (referred to as Test-4 and Test-6) to evaluate the model's performance when there 4 or 6 target images in videos.
18

---

matrix fol- Published as a conference paper at ICLR 2023
Table 4: The hyperparameters of CLAP-NP.
Dataset
|C|
B.
B
3TC
02
dg
BoBa-1
100
100
1000
0.03
BoBa-2
6
400
100
1500
0.03
64
CRPM-T
400
100
5000
0.1
CRPM-DT
3
200
50
5000
0.1
64
CRPM-C
100
100
5000
0.1
CRPM-DC
3
300
300
10000
0.1
64
MPI3D
50
50
150
0.03
Continuous Raven's Progressive Matrix (CRPM) CRPM consists of 3 image matrices where the images contain one or two centered triangles or circles. CRPM provides four instances with different image types: CRPM-T, CRPM-DT, CRPM-C, and CRPM-DC. Images low attribute-specifee changing rules (c.g., rules in the size, grayscale, and rotation of the triangle). The first sample in Figure 8c shows that, in the row of the matrix, the sizes of triangles increase pro- gressively, the grayscales change progressively from dark light, and the rotations keep constant. To predict the missing target images in the matrix, the models need to learn the probability space on functions {-1, 1)² + IR 64x64 that map 2D grid coordinates to grid images. Table 2 shows that we provide 10,000 image matrices for training. 1,000 for tuning the model hyperparameters, and 2,000 to test the abstract visual reasoning ability of models. In the training and validation process, we randomly select 1 or 2 images as target images; in the testing process, we use Test-2 and Test-3 evaluate the performance of the models predict 2 or 3 target images.
MPI3D MPI3D (Gondal et al. 2019) dataset contains a series of single-object scenes, each with 40 different scene images. There are two underlying rules to change the object in scene: change the altitude of the object (sample 1 in Figure Bg); change the azimuth of the object (sample 2 in Bigure8gg... The other attributes (e.g., object color, camera height, etc.) do not change within scene images. is essential for the models to grasp different changing patterns for target image prediction, and the key to describe the distribution over functions f - R3x61161 that map object altitudes azimuths to scene images. As Table 2 shows, provide 16127 scenes for training, 2305 for tuning the hyperparameters, and 4608 for testing. terms of training and validation, we first randomly select images from 40 scene images represent the scene, and then randomly select to 4 images from the 8 images again as the target images, leaving the remaining images as the context. For testing, supply Test-4, Test-6, Test-10, and Test-20, where Test-4 and Test-6 evaluate the performance of the models to predict 4 or 6 target images out of 8 images; Test- 10 and Test-20 evaluate the performance predict 10 20 target images out of 40 images. One can fetch MPI3D from the repository under the Creative Commons Attribution 4.0 International License.
D.2 MODEL ARCHITECTURE AND HYPERPARAMETERS
CLAP-NP In this subsection, we will first describe the architecture of the encoder, decoder, concept-specific function parsers, and concept-specific target predictors CLAP-NP, Then we will list hyperparameters in the training phase.
Encoder: for all datasets, CLAP-NP uses the same convolutional blocks downsample high- dimensional images into the representations of concepts, which can be described as
4 x 4 Cony, stride 2, padding 1, 32 BatchNorm, ReLU
4 x 4 Cony, stride 2, padding 32 BatchNorm, ReLU
4 X 4 Cony, stride 2, padding 64 BatchNorm. ReLU
- X 4 Cony, stride 2, padding 1, 128 BatchNorm, ReLU
4 X 4 Conv, 256 BatchNorm, ReL.U
X 1 Cony, 512
- ReshaaeBlock, 512
hhtg:///gtmhhcmmmmmmrmmmmmiiiiim dataset
19

---

Published as a conference paper at ICLR 2023
Fully Connected, |C|
The ReshapeBlock flattens the output tensor of size 512 x X from the last convolutional layer the vector of size 512. The output of the encoder is the mean of CI concepts.
Decoder: CLAP-NP uses several deconvolutional layers upsample the representations of con-
cepts to images, which is
ReshapeBlock, |C] x x
- x 1 Deconv, 128 BatchNorm, ReLU
4 X 4 Deconv, 64 BatchNorm22, ReLU
4 X 4 Deconv, stride 2, padding 1, 32 BatchNorm, ReLU
4 X 4 Deconv, stride 2, padding .2 BatchNorm, ReLU
4 X 4 Deconv, stride 2. padding 32 BatchNorm. ReLU
x 4 Deconv, stride 2, padding NNChnnnl Sigmoid
where NChannel is the number of image channels and Sigmoid is the activation function used to generate pixel values ranging in (0, 1).
Function Parser: CLAP-NP provides identical but independent function parsers for concepts.
Each function parser consists of the network T, encode the representations of concepts, Tú to encode the inputs of latent random functions, Ta to aggregate context information, and Ty to estimate the distribution of the global latent variable. The architecture of T, and Ti is
Fully Connected, 32 ReLU
Fully Connected, 16
By concatenating the representations of concepts and the function inputs, we get the context points in the latent random function. The architecture of Ta is
Fully Connected, 256 ReLU
Fully Connected, 256 ReLU
Fully Connected, 128
Finally, the architecture of Ty is
Fully Connected, 256 ReLU
Fully Connected, 2dg
where dy is the size of the global latent variable gº. The outputs of Ty consist of the mean and standard deviation of gf
the concept-specific target predictor Tp maps gf and the encoded function inputs into the target concept, whose architecture is
Target Predictor:
Fully Connected, 256 ReLU
Fully Connected, 256 ReLU
Fully Connected,
To generate complex scene images in MPI3D, CLAP-NP uses deeper decoder with hidden sizes [512, 256, 128, 64, 32]. And we use Ty with hidden sizes [128, 128] and output size 64, T, with hidden size [64], and T, with hidden sizes [128, 128]. Hyperparameters for CLAP-NP on set the learning rate as 3 x 10-4 batch size 0.1. For MPI3D, we anneal 3r and By in the first 400 epochs. After each training compute the evidence lower bound (ELBO) of the current model and save the model with the largest ELBO as the trained model. For all datasets, CLAP-NP uses the Adam (Kingma Ba 2015) optimizer update parameters.
different datasets are shown in Table In all datasets, as 512, and the validation
epoch,
Neural Process (NP) To deal with high-dimensional images in NP (Garnelo et al. 2018b), we apply the encoder and decoder of CLAP-NP in it to convert high-dimensional images into low- dimensional representations. The aggregator Ta extract context information from the representations and function inputs. Then the function parser T, converts the context representation into the mean and standard deviation of the global latent variable. Finally, the decoder generates target images from the target function inputs and the global latent variable. The architectures of T, and Ty are
Aggregator
20

---

Published as a conference paper at ICLR 2023
Table 5: The hyperparameters NP.
Dataset
d
dg
BoBa-1
1024
BoBa-2
6
1024
CRPM-T
1024
CRPM-DT
3
1024
CRPM-C
512
CRPM-DC
3
512
MPI3D
1024
Table 6: The learning rate and training epoch of GP.
Dataset
learning rate
epoch
BoBa-1
0.05
200
BoBa-2
0.01
200
CRPM-T
0.05
200
CRPM-DT
0.05
100
CRPM-C
0.01
400
CRPM-DC
0.05
100
MPI3D
0.05
10
Fully Connected, 512 ReLU
Fully Connected, 512 ReLU
Fully Connected, 512
Function Parser
Fully Connected, 512 ReLU
Fully Connected, 512 ReLU
Fully Connected, 2d,
NP's hyperparameters on different datasets are shown in Table where d, is the representation size of the encoder. In all datasets, NP adopts the learning rate 10 batch size 512, and o, = 0.1. The parameters of NP are updated with the Adam optimizer KKngmma & Ba 2015).
Generative Query Network (GQN) To implement GQN (Eslami et al. 2018). we use a
P PyTorch implementation from the repository with the following changes to the default configuration to
control the computational resource: set learning rate to 0.0005, batch size to 256, the representation type to pool, the number of iterations to 4, and share the ConvLSTM core among iterations.
Gaussian Process (GP) We use pretrained autoencoder to reduce the dimension of images and let GP capture the changing patterns in the lowddimessional space. We adopt the same encoder and decoder as in CLAP-NP and NP for a fair comparison. The mean function of GP is set to a constant function. After comparing the RBF kemel, periodic kernel, and deep kernel, we chose the deep kernel as the kernel function that x( = o² exp(-I]TT.<;;; 2016). Neural network Ty a multilayer perceptron with hidden sizes [1000, 1000, 500, 50], output size 2, and ReLU activation functions, which is the same as in DKL (Wilson et al.) 2016). The hyperparameters of GP are tuned on each sample through the log-likelihood of context points and the R.Spropp optimizer. We give the learning rate and epoch to adjust hyperparameters in Table c and use multi-task Gaussian Process prediction (Bonilla al.) 2007) model the correlations between dimensions.
Tx (olz;)l///// 2 (Wilson et al.
21

---

Published as a conference paper at ICLR 2023
GT
XP
GP
00N
CLAPNP
GT
GP
CLAPNP
BoBa-1
GT
000
CLAP
GT
GP
C0N
CLAP-NP
(b) BoBa-2
Figure 9: Target prediction BoBa. The images with red boxes are predictions from the models.
D.3 COMPUTATIONAL RESOURCE
We train our model and baselines on the server with Intel(R) Xeon(R) Gold 6133 CPUs, 24GB NVIDIA GeForce RTX 3090 GPUs, 512GB RAM, and Ubuntu 18.04 OS. All models are imple- mented with the PyTorch (Paszke al. 2019) framework.
E ADDITIONAL EXPERIMENTAL RESULTS
E. INTUITIVE PHYSICS
In Figure 9 we display additional prediction results on BoBa-1 and BoBa-2. For each instance, we provide samples using the configurations Test-4 (left) or Test-6 (right), where CLAP-NP outper- forms other baselines. NP generates blurred target images on both BoBa-l and BoBa-2 datasets. indicates that NP has difficulty modeling changing patterns on bouncing balls. GQN can pro- duce clear images but may generate non-existent balls and lose existing balls. In the first sample of BoBa-I, 1. the predictions of GQN deviate from the ground truths significantly in the position balls. CLAP-NP performs well in modeling scene consistency and predicting motion trajectory.
22

---

Published as a conference paper at ICLR 2023
CLASS
(a) CRPM-T
(b) CRPM-DT
(c) CRPM-C
(d) CRPM-DC
Figure 10: Target predictions CRPM. For each sample, show two predictions of the models to test the understanding of attribute-specific rules.
E.2 ABSTRACT VISUAL REASONING
With the additional experimental results in Figure T0 we can intuitively see that CLAP-NP has better understanding of the changing rules than baselines. For the second sample of each dataset, row of images removed to interpret the abstract visual reasoning ability of the model. As samples in Figure 10 shows, when remove row from the matrix, the answer not unique: the predictions are correct as long as their sizes increase progressively and their colors and rotations keep constant. CLAP-NP represents such randomness in predictions by means of the probabilistic generative pro- cess, making possible to generate different correct answers. In terms of the prediction quality, although generating blurred images, NP has the basic reasoning ability about the progressive rule of
23

---

Published as a conference paper at ICLR 2023
Table 7: MSE scores on CRPM-T and CRPM-C.
CRPM-T (17 1~2/9)
CRPM-C (1= 1~2/9)
Model
1 2/9
3/9
1)= 2/9
1/ 3/9
NP
125.8 1: 4.9
204.6 1 5.9
112.8 1 6.9
186.8 1 12.6
GP
416.4 +: 44.2
611.0 + 44,9
286.0 26.5
432.1 1: 27.4
GQN
235.2 = 4.7
493.4 - 7.2
246.5 = 5.2
484.6 = 8.2
CLAP-NP
41.3 1.8
13.7 89.1
52.4 11.3
136.6 = 15.8
4P
00
Figure 11: SA scores CRPM-T and CRPM-C.
the outer triangle size in Figure 10b GQN generates clear images, however, the generated images probably deviate from the underlying changing rules on matrices. Table and Table T1 illustrate the outperforming results of CLAP-NP in abstract visual reasoning by quantitative experiments.
E.3 SCENE REPRESENTATION
Figure 12] shows the prediction results within 8 scene images. Both GQN and CLAP-NP generate clear prediction results when the number of target images is 4; if the number of target images is increased to 6, the prediction accuracy of GQN has a significant decline, while CLAP-NP maintains the accuracy. NP generates ambiguous results in all experiments. Figure 1 shows
com- plicated situation: we provide 40 scene images and set 20 of them as target images. In this case, GQN and NP can hardly generate clear foreground objects; while CLAP-NP produces relatively curate predictions with only a little decrease in generative quality. This experiment aims to test the generalization of the laws learned by the models, and the results above illustrate that NP can hardly represent scenes with functions, GQN has difficulty generalizing the scene representation ability to different configurations, and the laws learned by CLAP-NP can be generalized novel scenes with the compositional modeling of laws.
E.4 CONCEPT DECOMPOSITION
Concept decomposition is the foundation for CLAP understand concept-specific laws. In this experiment, we traverse each concept in the latent space and visualize the concepts through the decoder to illustrate the LRFs. First of all, encoder to estimate the range of concepts in the latent space. To traverse one concept, we fix the other concepts and linearly interpolate it from the minimum value the maximum value generate a sequence of interpolation results, which are decoded into images for visualization. Each row of Figure 14 represents the traversal results of one concept. In BoBa-1, CLAP-NP learns LRFs on concepts of color, horizontal position, and vertical position in an unsupervised manner. This similar to the way we understand the motion of balls: the color keeps constant over time; the horizontal and vertical positions conform to the physical laws in their respective directions. For CRPM, CLAP-NP correctly parses images into concepts that correspond to the stribute-speciife rules in matrices. Concept decomposition in real environments is challenge for models. For MPI3D, CLAP-NP parses the LRFs on the object appearance (Dimension 1, 5, 6, 7) and other static scene attributes (Dimension 2, 4).
decompose batch of images into concepts by the
24

---

Published as a conference paper at ICLR 2023
NP
GT
GP
02N
CLAPNP
Figure 12: Target prediction with 8 scene images on MPI3D
E.5 LATENT RANDOM FUNCTION COMPOSITION
By means of concept decomposition, CLAP-NP can parse the LRF on each concept to represent concept-specific laws, by which the law of sample can be modified in terms of concepts. Figure T6 display the way to manipulate the laws. We compose the concept-specific LRFs of existing samples generate samples that have novel changing patterns. For example, in the third sample of Figure 15a we generate a sample with the progressive law of size, constant law of rotation, and constant law of color from three existing samples. The results on four instances of CRPM indicate that CLAP-NP understands the changing patterns matrices in an interpretable and manipulable way, which embodies the compositionality of laws.
E.6 LATENT RANDOM FUNCTION EXCHANGE
Another way to manipulate laws is to exchange LRFs of some concepts between samples. In BoBa, we swap the law of appearance or object motion between samples, and the results are shown in Figure 17 According to the concept decomposition results in Figure [4] we exchange the LRFs between samples on Dimension (Dimensions 2 and 3) to modify the motion of balls (the law of appearance) in BoBa-1. And exchange the LRFs between samples Dimensions 1 and 6 (Dimensions 2, 3, 4, and 5) to modify the motion of balls (the law of appearance) in BoBa-2. To exchange laws in MPI3D,
shown in Figure 18 we swap LRFs between samples on Dimensions 5, 6, and 7 to change the law of the object motion. By modifying laws on example of Figure 18 the changing pattern of the first sample becomes the horizontal rotation, and the second sample inherits the law of vertical rotation from the first sample, which illustrates CLAP-NP ability to generate new samples. The experiments on law exchange evidence CLAP-NP's interpretability and manipulability from another perspective.
E.7 GENERALIZATION ON UNSEEN CONCEPTS
We extend BoBa-2 to generate four datasets with novel laws: Novel-colors (BoBa-2-NC), Novel- shape (BoBa-2-NS), Without-ball-collsioon (BoBa-2-WBO), and With-gravity (BoBa-2-WGH We draw balls on BoBa-2-NC with unseen colors and replace the balls on BoBa-2-NS with squares without changing the laws of motion. For BoBa-2-WBO, we disable collisions between balls and re-
25

---

Published as a conference paper at ICLR 2023
GT
NP
GP
GQN
CLAP-NP
GQN
CLAP-NP
GT
GQN
CLAP-NP
Figure 13: Target prediction with 40 scene images on MPI3D
26

---

Published as a conference paper at ICLR 2023
Latent Traversal
25
0.64
30.12
(6) (3)
linet
13.28
0.70
0.78
Latent Trveesal
Sin
Latent Traversl
1.52
2.48
4.46
0.50
2.07
1.26
(d) MPP1
1.54
0.65
7.96
0.26
4.15
Figure 14: Latent traversal results and variance declines on BoBa-1, CRPM-DT, CRPM-DC, and MPI3D.
27

---

Published as a conference paper at ICLR 2023
Rutation Color 21
Rotation Color 3]
Culer 0
Coler 31
Stire Rotation Coler
$ Size 2 Rotation
Sizee Rotation Coler
Size Rotation
4
A
A
Siire Rotation
Size 2,
Stoee Rotation
Sire Rotation
(a) CRPM-T
!
AF
A
A
!
Inner Coler Retation
[ Laner Color 2, Inner Rotation 2
Laner Coler Inner Ratation 3,
lnner Color Rotation 3]
A
A
A
Inner Coler Inner Retacion 1. Ouser Size 1
{ Inner Inner Rotution Outer Sizee 2
Laner Inner Ratacion 3, Outer Size 3
Inner Color laner Rotatioa 2, Outer Size 3
A
A
A
P
A
!
A
!
A
e
Jnner Coler Jnner Recation I. Outer Size
laner Color 2, Inner Rotation Outer Size
Inner Coler Inner Rotation 3, Outer Size 3
Jnner Color laner Rotation 2, Outer Size 3
(b) CRPM-DT
Figure 15: Law Composition CRPM-T and CRPM-DT.
28

---

Published as a conference paper at ICLR 2023
Secctor Size Sector Posicion 2
Sector Sector Position 3
Size 1,
Sectur Sector Position ]
Sector Position }
Sector Position 3
Sector Position 2
Secter Sector Position ]
Sector Size Sector Position
Sector Sizt Sector Position
Secter Size
Sector Size 2, Sector Position 2
Sector
Sector Position 3
Position )
I Sector Size I
I Sector Size 2, Sector Position 3
Sector Size 1, Sector Position 2
Sectior Size 1. Sector Position 1
Sector Size 2, Sector Position 2
I Sector Size Sector Position 2
Sector Position }
(a) CRPM-C
Inner Coler Sector Position 1. Secoor }
[ Inner Color 2, Sector Position 3.
Inner Color Sector Posirion 3, Sector 3
lnner Color Sector Podition 2, Sector Stee
Inner Coler Sector Podttio 1. Sector Sizee }
laner Sector Position Sector Size 2
Inner Coler Sector }
Inner Color Poditio 2, Sector Stre )
+
Inner Color I, Sector Podiiio I, Sector Size }
I Ianer Color 2. Sector Position 2, Sector Size 2
Inner Color Sector Posiiio 3, Sector Size 3
I Inner Color Sector Podition 2, Sector Siee 3
(b) CRPM-DC
Figure 16: Law Composition on CRPM-C and CRPM-DC.
29

---

Published as a conference paper at ICLR 2023
Collison Lan Apperrance Lanr
LRFs betweon sample
Eadwugg
Calinino Lam 2pparaacelan2
Collidin Law Apporance Lanc?;
Calliton Law 1pporaaceaand
(Colinins
Excharge BB betazan
(Celliies Le
(Ciliiio
sanpls Dan
92 and a3
[Cillion Apperseteecc)
(a) BoBa-1
Euduaue 1R04 howwan
Collidan Appirrance Lawe
Cxiliden
Apperranc Law
smmpla Din kl and do
[Calliin aa A'Apurrneet 1
[Calibinn Lalk Apperrmettaw
samples
[Calidan av Apuurance
[Cilldin 3aw 21
(b) BoBa-2
Figure 17: Law Exchange on BoBa.
30

---

Published as a conference paper at ICLR 2023
Sooe Law Objject Acta 13
LRFS beth
Fachunge
Pionn Law
D.jeeet Actie Law
(Seme Law 3, Object Lak 33
surples on Dan As, 25, 66, and it?
Soom Law 3, Ofjer aock Law
O.Ojjer Asdel
Exchunge 18T's betwezn
1.0kjer
(ooea Ofjjct Anke
surpiex Dén 85, 86, ut
Ofjerr Aoda
Figure 18: Law Exchange on MPI3D.
BoBa-2 (Novel-odoss)
BeBo-2 (Novcislupe)
BoBa-2 (W :olloco-
Bolla-2 graihyy
Figure 19: Evaluate CLAP-NP on BoBa-2 with novel colors, shapes, and physical laws.
31

---

Published as a conference paper at ICLR 2023
Table 8: MSE and SA scores on BoBa-2 with novel concepts or laws where the models are tested with the configuration = 12. SA-k indicates the SA scores with & candidates.
Metric
Model
BoBa-2-NC
BoBa-2-NS
BoBa-2-WBO
BoBa-2-WG
NP
2047.6 +: 10.9
990.2 + 14.1
1743.1 +: 15.7
1525,5 12,9
GP
2671.9 1 75.9
1907.4 1 74.2
2068.2 1: 124.9
2019.9 1 85.0
MSE
GQN
2350.8 1 55.4
1459.7 1 24.6
2365.8 1: 38.3
2092.9 + 23.5
CLAP-NP
1611.2 + 71.1
979.0 72.5
1420.3 + 60.4
1322.4 + 58.9
NP
0.984 1: 0.004
0.989 +: 0.003
0.925 1:: 0.010
0.956 + 0.008
GP
0.991 +: 0.005
0.986 + 0.009
0,974 1 0.011
0.985 1 0.012
SA-2
GQN
0.952 0.009
0.955 + 0.010
0.878 + 0.015
0.891 0.012
CLAP-NP
0.995 0.004
0.999 + 0.001
0.997 + 0.002
0.998 = 0.001
NP
0.972 0.005
0.981 + 0.004
0.876 + 0.018
0.924 + 0.005
GP
0.986 0.013
0.970 + 0.021
0.940 + 0.035
0.972 - 0.019
SA-3
GQN
0.920 + 0.007
0,926 + 0.012
0.821 + 0.009
0,820 = 0.015
CLAP-NP
0.990 + 0.006
0.998 + 0.002
0.995 + 0.005
0.998 + 0.002
NP
0.949 + 0.008
0.969 + 0.005
0.796 + 0.012
0,877 + 0.010
GP
0.982 1 0.012
0.962 1 0.021
0.907 1 0.049
0.968 +: 0.018
SA-5
GQN
0.884 0.013
0.887 + 0.010
0.734 + 0.028
0.741 = 0.013
CLAP-NP
0.978 0.016
0.997 + 0.003
0.991 + 0.006
0.995 - 0.004
NP
0.918 0.017
0.945 0.009
0.705 1 0.021
0.823 0.020
GP
0.954 0.031
0.948 + 0.021
0.879 = 0.038
0,946 = 0.028
SA-9
GQN
0.828 + 0.018
0,842 + 0.015
0.627 = 0.026
0,643 = 0.028
CLAP-NP
0.975 + 0.010
0.992 + 0.005
0.982 = 0.008
0.991 = 0.004
NP
0.884 + 0.015
0.917 = 0.011
0.584 + 0.026
0.752 + 0.019
GP
0.938 + 0.044
0.916 + 0.045
0,826 = 0.030
0.926 = 0.034
SA-17
GQN
0.781 0.023
0.782 0.026
0.549 + 0.014
0.558 0.016
CLAP-NP
0.965 0.014
0.991 + 0.006
0.972 + 0.007
0.982 + 0.013
serve collisions between balls and borders; for BoBa-2-WG we add vertical gravity scenes. After training on BoBa-2, CLAP-NP is tested on four datasets without retraining to evaluate whether the compositionality of laws improves the model's generalization ability in scenes with unseen concepts or laws. Figure (9uuree99ssoo shoss that CLAP-NP can predict the correct object positions on BoBa-2-NC and BoBa-2-NS. CLAP-NP predicts inaccurate color and shape of objects because the encoder and de- coder are trained on BoBa-2, and the latent space does not encode the unseen colors or shapes. We observe similar results on B-BO and 2-Ba-2-WG that CLAP-NP learns the correct law of appearance but predicts incorrect positions of balls when there are unseen physical laws in scenes. To better evaluate the generalization ability in scenes with novel laws, we quantitatively evaluate CLAP--P and baseline models with MSE ad SA scores on the four datasets. The results in Table g indicaae that CLAP-NP achieves the best MSE and SA scores in most situations, which illustrates CLAP-NP's generalization ability in scenes with novel concepts or laws.
E.8 NUMBER OF LATENT RANDOM FUNCTIONS
This experiment explores the influence of setting too many or too few LRFs in CLAP-NP. Figure 20 shows the latent traversal results on MP13D, BoBa-2, and CRPM-DT. we set too few LRFs, CLAP- NP encodes different laws in one LRF instead of learning compositional laws, which will influence CLAP-NP's generation ability (e.g., set only two LRFs for BoBa-2). Setting too many LRFs has no significant influence on CLAP-NP's performance because there will be redundant dimensions in CLAP-NP that do not encode information (e.g., dimensions 1, 3, and 6 on CRPM-DT). However, due the independent modeling of concept-specific LRFs, setting large number of LRFs will reduce the computational speed of CLAP-NP and increase the number of model parameters.
E.9 PREDICTION STRATEGY
CLAP uses the one-shot strategy that predicts all target images at one time. However, the rollout strategy be another choice to predict the following target images through the few context images
32

---

Published as a conference paper at ICLR 2023
B
CRPS-DI(CC1
Figure 20: Latent traversal results with too many (top) and too few (bottom) LRFs.
Table 9: MSE scores of CLAP-NP with the rollout and one-shot strategy.
Dataset
CRPM-DT (1) 6/9)
BoBa-2 (1) 6/12)
MPI3D (17 = 20/40)
Rollout
362.5 3.5
4651.8 15.7
228.2 + 0.6
One-shot
361.9 + 2.3
4778.5 + 15.8
237.3 + 1.2
the beginning. For example, in the test configuration 7 20/25, we first predict the 6th to 10th images from the first five context images, then combine them (Ist to 10th images) to predict the following five images, and repeat this process until all target images are predicted. Table shows the MSE scores on CRPM-DT, BoBa-2, and MPI3D where CLAP-NP predicts targets with the rollout and one-shot strategy, respectively. On BoBa-2 and MPI3D, the rollout strategy slightly improves the prediction accuracy. Generally, if a target image is far away from all the context images, the prediction may have high uncertainty. The model only predicts a few target images close to the context each time with the rollout strategy, while the one-shot strategy requires the model to predict all the target images at one time. Therefore, the rollout strategy is more likely to have lower computational uncertainty and higher prediction accuracy.
E. FAILURE CASES
In this experiment, we display some failure cases. For BoBa, most failure cases occur when there are continuous target images (Ist sample of BoBa in Figure 21} too many target images (2nd sample of BoBa in Figure 21). For CRPM, CLAP can predict target images with diversity when we remove row from a matrix, but sometimes generates target images that break the rules. For example, the color of images keeps invariant in the first two rows, but CLAP generates images in the third row with changing grayscales (samples of CRPM in Figure 2T). For MPI3D, when the object size has an obvious change (1st sample of MPI3D in Figure 21 the centric object is tiny (2nd sample of MP13D in Figure 2T), the predictions can be incorrect or unclear.
LIMITATIONS
We conclude our limitations in two aspects. (1) Complexity of datasets. Because compositional law parsing is an underexplored task, we first validate the effectiveness of CLAP on datasets with rela- tively clear and simple rules to avoid the influence unknown confounders in complicated datasets. We believe that the compositionality of laws also exists in more complex scenarios (e.g., learning physical laws realistic scenes) and some vision tasks may benefit from compositional law parsing. For example, can perform controllable video generation process based on law modification or make more interpretable predictions by analyzing dominant laws in videos. Discovering such com- positional law parsing ability in more complex situations be valuable topic in future works.
33

---

Published as a conference paper at ICLR 2023
RoRa-22
!
A
A
P
CRPM
A
A
!
Figure 21: Failure cases of CLAP-NP.
34

---

Published as a conference paper at ICLR 2023
(2) Setting the number of LRFs. CLAP places upper bound on the number of LRFs. For scenes with multiple complex rules, we can empirically set an appropriate upper bound or directly put a large upper bound for the number of LRFs. However, large bound will linearly increase the num- ber of model parameters, and the redundant concepts will waste computing resources, decreasing computational efficiency. Therefore, exploring mechanism in the future to dynamically adjust the number of functions will be meaningful. When applying CLAP to more complex scenes, we may introduce more inductive biases for better concept decomposition (e.g. use more task-specific en- coder or decoder). We design CLAP as an unsupervised model, which means that it can be extended take advantage of task-specific annotations. Thus we can find ways to integrate CLAP with ad- ditional supervision information (e.g., supervise CLAP with the changing factors of scenes) to help concept learning in a specific task.
REFERENCES
Edwin V Bonilla, Kian Chai, and Christopher Williams. Multi-task gaussian process prediction.
Advances in neural information processing systems, 20, 2007.
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disen- tanglement in variational autoencoders. In Advances in Neural Information Processing Systems, pp. 2610-2620, 2018
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, 360(6394): 2018.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv:1807.01622, 2018.
Muhammad Waleed Gondal, Manuel Wuthrich, Djordje Miladinovic, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation the real world: new disentanglement dataset. Advances in Neural Information Processing Systems, 32, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pp. 370-378. PMLR, 2016
35


Reviews:
[{"review_id": "cnVunz9mwd", "reviewer": "Reviewer_i5ZN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a model for compositional law parsing that is capable to accurately represent different \"laws\" of semantic concepts. The laws are built with random functions, and the proposed system can combine these functions to achieve flexibility and generality in different law parsing tasks.\n\nThe model uses an encoder-decoder architecture that represents concepts as latent variables, and laws are operationalized as concept-specific functions built with neural processes.\n\nThe authors claim that the proposed framework outperforms other methods in scene representation, intuitive physics and abstract visual reasoning tasks. ", "summary_of_the_review": "The paper introduces a novel and powerful concept. The experiments are thorough and the mathematical formulation appears to be thoroughly supported. Its main weakness is tied to clarity: the manuscript assumes large amount of previous knowledge related to NP and other topics, and clarity could be improved to ensure more fluid reading. In general, the paper establishes a powerful framework with strong improved performance over baselines over several benchmarks.", "strength_and_weaknesses": "## Strengths\n- Powerful concept with several meaningful abstractions\n- In-depth mathematical formulation explaining the formalities of the proposed framework\n- Detailed experiments, including multiple different benchmarks with very different setups and objectives\n- Very strong performance against baselines\n\n## Weaknesses\n- The abstract is not very good at exemplifying the proposed framework. More information, and possibly examples, describing what each law and concept can be would be great so that the concept becomes clear from the get-go.\n- The explanation of the Neural Process in Section 3 is hard to understand. Clarifying that paragraph might be of great help to readers unfamiliar with NPs.\n- General clarity. The paper is not easy to follow without substantial previous knowledge of the field, and it is hard to judge the \n- Baselines used: the set of baselines used is small, and an inclusion of other works could improve the experiments section. Importantly, comparing to task-specific networks might be interesting to observe how this general framework competes. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "cJWSL_gt3v", "reviewer": "Reviewer_Fbxt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper brings compositionality into the Neural Process framework. It decomposes the input into multiple concepts, each represented by a latent vector. A Neural Process is learned for each concept to capture its law, so that the concept can be generated at query points given a few context points. The full pipeline is trained end-to-end via variational inference without supervision. Experiments on three diverse datasets show that the proposed model can produce more accurate pixel predictions and more consistent latent space than NP and GQN. It is also qualitatively shown that one can compose different laws by combining the corresponding neural processes.", "summary_of_the_review": "I am leaning toward reject for the paper in its current form, mainly because the novelty needs more clarification.", "strength_and_weaknesses": "- Strengths\n    - The experiment results are quite extensive. The failure of NP and GQN is clearly shown. The law manipulation result (Figure 7) is very interesting and shows that the learned neural processes can indeed be composed.\n    - The paper is well organized, and introduces sufficient background knowledge for readers to follow.\n- Weaknesses\n    - At a high level, the proposed method is similar to [ROOTS](https://www.jmlr.org/papers/v22/20-1176.html), which brings compositionality into GQN and focuses on compositional understanding of 3D scenes. Both of them can be thought of as a compositional NP framework. ROOTS has similarly shown better generation quality than GQN, and can compose novel scenes by combining the object-level GQNs. However, I think there is still merit in this paper, as it seems more general and not limited to 3D scenes. I encourage the authors to highlight their novelty compared to ROOTS.\n    - The datasets look visually simple. Some compositional video scene understanding methods (e.g., Kosiorek et al., 2018; Jiang et al., 2019; Lin et al., 2020a as listed in related work) can deal with more complex scenes. Can you clarify why they are not considered baselines?\n- Questions and Minor Issues\n    - How many concepts did you use and what is the dimension of the latent vector for each concept?\n    - The bouncing ball dataset is sometimes referred to as BoBa and sometimes BaBo.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_uetkGqS6ty", "reviewer": "Reviewer_HFjB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a latent variable model that parses dynamic scenes into independent laws represented as neural random functions. The model is trained to reconstruct the image sequence while keeping the distribution of concepts and random functions not changing over time and concepts. The evaluation on various datasets shows that it can predict the target images well. The qualitative examples show that the decomposed laws correlate with the changing visual concepts and can be swapped to generate novel samples.", "summary_of_the_review": "The paper proposes a novel approach to interpret dynamic scenes. It is well presented and easy to follow. It provides several examples of learned compositional laws. It will be much convincing if the authors can provide quantitative evaluation of compositional generalization.", "strength_and_weaknesses": "Strength\n- This is an interesting paper that focuses on learning disentangled laws to interpret dynamic scenes.\n- This paper also proposes a novel approach to learning concept-specific laws with Neural Proces. \n- The paper shows several qualitative examples to show how the results are interpretable and the composition of learned latent functions is meaningful.\n\nWeaknesses\n- It is unclear how the authors determined the number of laws for learning. What if we choose too many/little laws? Or how many laws we may need to interpret a realistic video?\n- The current quantitative evaluation focuses on MLE and selection accuracy to show the improvement of semantic error but these two don’t really represent the model learns the right laws. The MLE can be a result of better visual encoding, the selection accuracy shows better representation. If the model learns compositional laws, it can generalize better to novel combinations of concepts that are not in training, e.g. new colors. The paper currently only shows one generalization example in the appendix. The evaluation will be much stronger to show a quantitative evaluation of generalization capability.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Xzfz4lZ254L", "reviewer": "Reviewer_rEh1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a latent variable model to capture concept-specific laws from images/videos. The laws are encoded as Latent Random Functions (specifically, Neural Processes), and can be composed with one another. ", "summary_of_the_review": "The following questions will help me assess the work better:\n- Is anything preventing you from __assessing__ the __disentanglement__ of the LRFs? It would be useful to have those metrics (e.g. see https://github.com/google-research/disentanglement_lib). \n- What would be __other potential realizations__ of CLAP which don’t use Neural Processes?\n- __Future work__: how do you expect LRFs would perform on the [Abstraction and Reasoning Challenge](https://www.kaggle.com/c/abstraction-and-reasoning-challenge)? Would that be an ideal target domain?\n- __How to generate future predictions__: You could generate an extended rollout (e.g. on BoBa) by applying the model autoregressively on the most recent generated frames. For example: to generate 20 target frames after 5 context frames, you could first generate frames 6-10, then use those to generate 11-15, etc. Alternatively, you could evaluate one-shot using a \\eta=20/25. Which one do you expect would work better? The former strategy would allow you to re-infer the concepts and global latent at each application.\n- On specific figures:\n  - Figure 7: The newly generated example on CRPM-T doesn’t have the same color law as matrix 3 right? I expected the triangles in the top row to be dark and identically colored. Rather, the triangles in the top row seem to be following a pattern that is different from all other top rows (from the context image). How is that possible?\n  - Figures 4 and 5: could you confirm you’re using a single model on each dataset (but multiple test configurations)?\n  - Figure 14: do you learn any latents on CRPM-DT which change the size/rotation of the outer triangle?", "strength_and_weaknesses": "Strengths:\n- Multiple datasets with different focus\n- Illustration of compositionalilty\n- Demonstration of failure cases\n\nCons:\n- Choice of baseline: neither NP nor GQN seem like strong, obvious baselines. Wouldn’t a system like [Aloe (Ding et al. 2020)](https://arxiv.org/abs/2012.08508) offer a better comparison?\n- Based on Tables 3 and 4, I worry that the model needs careful tuning to work or several runs for a winner to emerge. The best hyperparameters vary significantly across datasets. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7noJBPbx0A", "reviewer": "Reviewer_4jVq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Paper proposes a novel method to decompose a scene into representation of concepts such as object appearance, background, and angle of object rotation. This is done through an encoding-decoding architecture which represents the concepts as latent variables. These concepts are then recomposite via a decoder network to generate the target images.\n\nPaper further proposes to use Neural Processes (CLAP-NP) to instantiated each learnt concepts for inference purposes. Different experiments were ran to validate the model performance against baseline models for the intuitive physics, abstract visual reasoning and scene representation tasks. The proposed method significantly overperforms all baseline methods with the MSE and new derived metric, Selection Accuracy (SA).\n\nFinally, paper also shows that the method can be used to edit the individual concepts to generate new samples.", "summary_of_the_review": "Overall, this work seems excellent and of high impact. However, due to my unfamiliar with the technique described, it is difficult for me to review this work properly.\n\nThe experimental results are convincing. But I note that there is no comparison with any SOTA methods for tasks.", "strength_and_weaknesses": "Strengths\n1. Paper is well-written and well-motivated.\n2. The concepts are clearly explained with formulas and diagrams\n3. Claims are sufficiently supported by experiments.\n\nWeaknesses\n1. Paper introduces several concepts which are not mainstream. However, this is rectified by providing the reader with references and extensive Appendix material.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster