Relational Learning with Variational Bayes
Authors: Kuang-Hung Liu
Venue:  (2021)
Keywords: Relational learning
unsupervised learning
variational inference
probabilistic graphical model

Abstract: In psychology, relational learning refers to the ability to recognize and respond to
relationship among objects irrespective of the nature of those objects. Relational
learning has long been recognized as a hallmark of human cognition and a key
question in artificial intelligence research. In this work, we propose an unsupervised
learning method for addressing the relational learning problem where we
learn the underlying relationship between a pair of data irrespective of the nature
of those data. The central idea of the proposed method is to encapsulate the relational
learning problem with a probabilistic graphical model in which we perform
inference to learn about data relationships and other relational processing tasks.

Under review as a conference paper at ICLR 2021
RELATIONAL LEARNING WITH VARIATIONAL BAYES
Anonymous authors Paper under double-blind review
ABSTRACT
In psychology, relational learning refers to the ability to recognize and respond relationship among objects irrespective of the nature of those objects. Relational learning has long been recognized as a hallmark of human cognition and key question in artificial intelligence research. In this work, we propose unsuper- vised learning method for addressing the relational learning problem where we learn the underlying relationship between pair of data irrespective of the nature of those data. The central idea of the proposed method is to encapsulate the rela- tional learning problem with a probabilistic graphical model in which we perform inference to learn about data relationships and other relational processing tasks.
INTRODUCTION
American Psychological Association defines relational learning as (VandenBos & APA, 2007 Definition 1.1 (Relational learning). Learning to differentiate among stimuli on the basis of rela- tional properties rather than absolute properties.
In other words, relational learning refers the ability recognize and respond relationship (called relational property) among objects irrespective of the nature of those objects (called absolute property). Relational learning has long been recognized
hallmark of human cognition, and there has been substantial research showing that adequate cognitive capacity is necessary for relational processing (Biederman) 1987 Medin et al. 1993; Holyoak] 2012, Doumas & Hummel 2013
Gentner 2016). As a machine learning application, relational learning can provide new insight into data analysis by dissecting information the data into relational property and absolute property. However, in order to discover relationship patterns among raw and unknown data, relational learning only truly useful it can be achieved without supervised data. key challenge learning relational property with machine learning-biis baasdd based methods is that relational property is an abstract construct; unlike absolute property, which based on observable data and can be quantitatively measured, relational property is an abstract quantity that is difficult to objectively quantify, especially when the learning is unsupervised.
In this work, we propose an unsupervised learning vaaiittinna relation learning (VRL) for addressing the relational learning problem. The proposed method is completely unsupervised, which means that the learning does not require labeled training dataset training examples that have the same (or different) relational property. At its core, VRL encapsulates the relational learning problem with probabilistic graphical model (PGM) in which we perform inference to learn about relational property and other relational processing tasks. Furthermore, our main learning algorithm derived from the PGM using first principles, which gives the flexibility to use any compatible computational inference method and still retains the desired properties of the proposed method. Our contribution in this paper is threefold. First, we propose PGM that encapsulates the relational learning problem. Second, we formulate various relational processing tasks as performing inference and learning the PGM. Third, we propose an efficient and effective learning algorithm that can be trained end-to-end and unsupervised.
PROBLEM DEFINITION
We begin with formulating the relational learning problem as a machine learning problem: we observed paired dataset X { (a((), b))) | i [1..N] consisting of N i.i.d samples generated

---

Under review as a conference paper at ICLR 2021
from joint distribution p( a A, b B); our goal is to leam relational property between a and b(O) irrespective of their absolute property. Furthermore, want the learning to be unsupervised, e.g., we do not require a labeled dataset, such as (a b(t) 2()) where 2(1) is target variable indicating (a()) b ys relational property, nor do we require training examples that have the same (or different) relational property. There are two distinct features that separate our problem formulation from other unsupervised learning problem formulations:
1. We dissect the information in X into relational property and absolute property; relational property characterizes the relationship between a) and b( whereas absolute property represents specific features that independently describe ( and b(8)
2. Our goal is learn relational property among X irrespective of its absolute property, i.e., we want to learn a relational property that is decoupled from the absolute property.
In addition, we are interested in two related relational processing tasks: relational discrimination! and relational mapping VandenBoos & APA,
discrimination allows differentiate (a(*) (()) from (a 3, b based on their relational properties, while relational mapping allows us to apply the relational property of (a b()) to a different set of data, for example, deduce that (
is related to a()) in the same way that ((0) related to (*)
3 METHOD
Here introduce the proposed VRL method for addressing the relational learning problem and discuss various optimization challenges unique to VRL.
3.1
VARIATIONAL RELATION LEARNING
The proposed VRL method consists of two parts: first, we encapsulate the relational learning problem with PGM, called VRL-PGM; we then formulate various relational processing tasks as performing inference and learning in VRL-PGM.
b
N
Figure 1: VRL-PGM: probabilistic graphical model for representing the relational learning problem; the observed random variables a and are generated from some random process (parameterized by 0) involving unobserved random variable
The VRL-PGM model, shown in Fig. ! generates data z, and b by sampling from PDFs that come from parametric families distributions- -oo a ).pPo(( 2), Po b b|a, tha are differentiable almost everywhere with respect (w.r.t.) a, 2, and 0. In practice, realizations (a(), ((0) i [1..N]] while the true parameter 0* and the corresponding latent variables (*) unobserved. A well-known property of the PGM shown in Fig. [ is that random variables a and z are independent with no variables observed, but not conditionally independent when b is observed, i.e., po( a, )= = po( a )po( ), po b A po a b )po( b) (Bishop) 20066). In VRL-PGM, the absolute property can be interpreted as representing the dependency between a and
observe only set of independent
Definition (Relational discrimination in condition). discrimination based the relationship between or among stimuli rather than on absolute features of the stimuli.
Definition (Relational mapping) The ability apply what one knows about one set of elements to different of elements.
2

---

Under review as a conference paper at ICLR 2021
b, i..., features in a that can be used to predict b, while the relational property, represented by the latent variable z, can be interpreted as any additional information not found in a but can help to better predict b. The key requirement that the learned relational property be decoupled from the absolute property is enforced by VRL-PGM'ss construction where z, which represents the relational property, and a, from which the absolute property is derived, reflects our priority and compromise for using a PGM to represent the abstract relational learning problem: we sacrifice some identifiability of the original abstract problem (e.g.. VRL-PGM artificailly introduces causal relationship between a, b and b) but we gained rigorous and mathematical tractable PGM while achieving our primary objective of learning a decoupled (independent) relational property. Additional discussions on the connection between VRL-PGM and the relational learning problem is provided in appendix D.I
independent. The proposed VRL-PGM
Having established VRL-PGM, our primary learning objective is to approximate the unknown true likelihood function po b a, z) and posterior Po z a, b) (note that by observing b, random variables z and a are no longer independent). Learning po relational property (9) moreover, serves as a basis for performing relational discrimination where we compare relational properties between different pairs of data. Learning Pa( b a, 2. allows us to perform relational mapping where we use the relational property of (a(*), b(0)) to map from ( to
a, b) provides us a way to infer (a(*) b(')'ss
b(,), i.e., bó) po( b a(i), 2(() where 2(() po 2 a()) b(()
3.2 VARIATIONAL LOWER BOUND
We estimate the parameter for po b a, z by following the Bexiimum-likelioooe (ML) principle, and
approximate the true posterior Po a, b with variational Bayesian approach. More specifically, we use a variational distribution do( a, b parameterized by é, to approximate the unknown (and
often intractable) true posterior through maximizing variational lower bound (Bishopl 2006). To derive such lower bound, we first write the log-evidence as log po X) log po {a b() iE
[L...N]} Eiz log Po (1), ((6)) ). where each term in the summation can be expressed as:
log log (1)) (b()) DKL (go( z (1))) ((())))) po a (1) (.)(())))))
E Ro log z, (1) (6) log to z al), b())
(1)
The first term on the RHS is the KL-divergence from po z a (1)) (+)) to go z a()) b ). which provides measure of dissimilarity between the two distributions; the second term on the RHS continues as:
E (2la())))))
log po z, a...b.....) )- - log ( al.
(()))
log Pa (0) |a ((()) 2))p(( z ( a (1) log gol a (b))))))) b]
R (2)aa(())) log P (+) |a()) 2.) + log po 2) log 1s 2
(1) }] + log pe
log (1)),, (a
(2)
where in the second line we use the fact that random variables a and z are independent. Substitute Eq. back in ( and rearrange terms gives us:
log b ( aa(()) DKL (go(z) ())())))))))))))))) po(z (())) + [(0, oi a (())
(3)
where
(0) = E (gg(tla() [log po (0) 11) (0) (())),)) log log qo( (b))),)))))) (4)
C(0, di
Since KI.-divergence is non-negative, L(0, Φi (+) b))) (abbreviated as (() for notation compact- ness) serves as a lower bound for the conditional likelihod log Pa
(+) a))) ). Maximizing (
w.r.t. @ and 0 gives us both a ML estimate for pe( b a, z) (by maximizing the first term inside the
expectation in Eq. (4) and lower KI KL-divergence (the better go z ( (() approximates the true
posterior p z a (8) (()) )) as the conditional log-likelihood log po( b00 (6) does not depend on The lower bound c() can be maximized with gradient ascend methods; however, its gradients w.r.t. is difficult obtain: the expectation in Eq. 4 is taken w.r.t. the distribution qa( z (1)), b(t) ), which
3

---

Under review as a conference paper at ICLR 2021
is a function of Φ (Paisley al. 2012). To obtain efficient estimators for both ( and its gradients,
we adopt the reparameterization trick developed in Kingma Welling (2014) where the random
variable is expressed
transformation of another random variable p E that is independent
of a, b, and d: g(c, ( b(), x) where g is some differentiable and invertible transformation.
Given such change of variable, the lower bound (()) can be rewritten as:
( = Ep(e) log pe b (())
(1) (,))) + log ( z log g0( a (
(5)
where g(e, a (,) b(1) 6) and E p( E). Note that the expectation in Eq. (5) is taken w.r.t p(e) and we can now approximate (6) with Monte Carlo estimator:
L
2
I
1
Σ
log P b( a a (..)) ) +logPo( ( - log gol (z a b(o),
(6)
1=1
where (1.0) g(eli.)) a() b(0, Ø) and (6.0) p(E). The lower bound for a minibatches of data
N. XM = (a b()) iE [E.. can be approximated by ((0.xxx)) = M E 2(0) And finally, the gradients Vo,e Φ: x=)=
M Vo. can be computed in straightforward manner and used to update the parameters and Φ with stochastic optimization methods, such as SGD. Finally, additional discussion provided in appendix D where we explain how the optimization of the variational lower bound in Eq. [ naturally encourages the indpendence of z and a; we also discuss possible extensions to Eq. to explicitly safeguard against introducing dependency between and
3.3 OPTIMIZATION CHALLENGES
The proposed VRL method introduces unique challenges to the variational lower bound optimization problem (see Sønderby et al. (20T6) and Bowman et al. (2016) for other known challenges). To explain these challenges, steps (using a single datapoint as an example): (1) a datapoint (a(*), b())
first break down VRL's parameter updating process into the following ) is selected; (2) sample (++) To. 2 a ((0)) by using the current parameter Φu; (3) evaluate [() by using @x. Ggg (4) calculate gradients g = Ve..o .o. (()); (5) use gradients g to update Φx. 0g and get new parameters Φx+1, 0x+1. This parameter updating process can be depicted with an information flow diagram shown in Fig. 2a Ideally, we would like every path in Fig. ta contribute to the evaluation of all the terms in its reachable nodes in order obtain meaningful gradients for updating its associated parameters; however, there are two situations where this is not the case. The first situation, called
(1)
b (
a (5)
b(
(9)
(i)
go (z)
a
(
())
(1)
(1)
qo (z) a
(1)
.bb b(())
9
0
Pa(b (9) la z )), pa (2)
pa (b (1) a (()), (x(
(a)
(b)
Figure 2: Information flow diagrams depicting VRL's parameter updating process, where each path uses its associated parameters to propagate information in the forward direction and gradients in the backward direction: (a) Unmodified parameter updating process, where overfitting occurs when or the
the learning of pa (b ( 2()) rely only on the dashedotted path (deterministic-mang dashed path (information-ssoottaal (b) Parameter updating process improved with RPDA.
information-ssorttin occurs when the learning of po (+) a (0)) rely entirely on the dashed path in Fig. 2a more specifically, the dashed (shorteut) path directly propagates b(0 through (
4

---

Under review as a conference paper at ICLR 2021
to po (b(1) 20,))(())))))) ) and, result, the relational property z( may learn only to encode the absolute property of ((t), The second situation, called determiinistic-maapiie
occurs when b() can be fully characterized by ( in this case, the learning of Po ((() a z( may rely only on the dash-dotted path in Fig. 2a While both situations can be viewed as a overfitting problem, deterministii-muppph
says more about the data itself and exposes a potential limitation of VRL: a decoupled relational property is primarily learned through exploring information not found a but can help to better predict b, and when a fully characterizes b, VRL no longer need to explore information beyond a to predict and this may prevent VRL from learning a meaningful relational property. On the other hand, information-shortaal is caused by short-cutting the parameter updating process, which we may overcome with additional regularization techniques.
Here propose approaches for mitigating the iitommation-shotetke problem by disrupting the flow of information passing through the shortcut path. In the first approach, we restrict the flow of information by constraining the expressiveness of the latent variable z; for example, by adopting an informative prior with restrictive constraint, such as pe z = (z: 0, I a < 1, or representing z with discrete random variable (assuming we know a priori the underlying relational proproty are discrete). In the second approach, we propose a novel data augmentation strategy relattoo- preserving data augmentation (RPDA) that aims to eliminate the shorteut path. First, we define set of relation preserving functions D d(a. b;r) r R(some index set), A B ~ 1BB-AA Ax where the data relationship is preserved in the following sense: pa z a, b) = pa z a', b') for (a', b') d(a, r) Vr. Assuming we have access to such D, the proposed RPDA strategy then seek to optimize a modified lower bound estimator 210 C
log ( ((9) 1a(),((1))) ((111)) ) log (2))) -log (x())) (a(0),((()))
RPDA
(7)
L 1=1
(ax br(i)) = d(a b(0); (0), ,00 ~ U(R),
where
and 2((,)) = ((E(t.)) (0))) b((0) .)) clii)) ~ x((E). Note that due to the relation preserving property of D. we have go( ((() (0)) br(i) = a(0) b(²) and, therefore, Z(i is equivalent to
260 in Eq. (6). When we optimize with CRPDA the parameter updating process can be redrawn in Fig. 2b where now the learning of po( ((0) a()), 2(()) can longer rely solely on the shortcut path to propagate b((+) since it differs from (() by a non-deterministic factor p(0) In practice, construct set of RPDA functions D without extensive knowledge of the underlying relational property. However, we can treat data augmentation as a form of regularization and construct D that reflects our prior knowledge and belief of the underlying system (Ronneberger et al. 2015; Perez& & Wang 2017). For example, if we want the learning to be rotation invariance (a common theme in computer vision applications), we can construct a D) that consists of image rotation augmentations, c.g., d(a, b;r) (rot a, r), rot (b, r)) where rot(x, rotates the image x by r E R = [0, 360) degrees (note that both a and b are rotated by the same amount). Additional remarks on the practical applicability of RPDA is provided in appendix D.3 and a detailed ablation study is provided in appendix C
may seem unrealistic to assume that
To summarize this section, the proposed VRI. method with RPDA is described in Algorithm
4 RELATED WORK
Machine learning approaches for relational processing have gained increasing interest and attention in the literature. Most of these methods focus on high-level cognitive tasks, such as visual Q&A A and state prediction for complex-physic systems, and derive their relational processing capabilities from learning with clever designed neural networks (Hill et al.] 2019 Santoro et al.] 2017 Raposo et al. 2017, Battaglia et al. 2018 2016 Wu et al., 2015; Reed et al. 2015 van Steenkiste et al. 2018 Chang et al. 2016 Fragkiadaki et al.) 2015). Our work differ from these methods in two ways: (1) our primary focus is addressing the relational learning problem where we want to learn a decoupled relational property: (2) enforce the decoupling requirement on the learned relational property with a PGM, which gives us the flexibility to use any compatible inference method or function approximation and still satisfy the decoupling requirement.
5

---

Under review as a conference paper at ICLR 2021
Algorithm 1 VRL with RPDA
procedure VRL(X, p(e), D)
> If not available, D= (id(-) ( (a,
id(a. b)
Initialize parameters 0
while not convergence of parameters (0, do
Sample minibatch I b() ie (1. } from X.
Sample e(!-1) p(e), r() ((),, i= M,
L.
Run RPDA and obtain (a b ((0) = (la()bb0rri
i= i= M.
Compute gradients g Vo,aC (0,
o; x) N E Vo,oCRrD (see Eq. 7
Update parameters & using gradients g (c.g. SGD).
end while
return 8,0 end procedure
Conventional unsupervised learning methods can also be appied to our problem setting (Kingma
& Welling 2014 Goodfellow al.) 2014 Mikolov al.) 2013a [Song et al.) (007); however, these methods learn single representation for the data with superimposing information about their relational and absolute property. The difficulty of decoupling the relational property from the learned representation constitute major obstacle to relational reasoning.
Other related work include methods on learning a disentangled representations with applications in style-transfer, image-to-image translation, domain adaptation, etc. (Huang et al.) [2018] Chen et al.
2016 Tenenbaum Freeman, 1997; Higgins et al. 2017 Bousmalis et al. [016]] Mathieu al. 2016 Tulyakov et al. 2017] Denton & Birodkar 201 Villegas et al. 2017 Donahue et al. 2017 content and style (or pose for video sequence data) where content is generically defined the underling spatial structure, and style as the rendering the structure. In comparison, our work can be viewed as learning disentangled representations of relational and absolute property; however, argue that style-content separation is fundamentally different from cicational-absolle separation. More specifically, we consider both style and content information absolute property (both deseribe features of an individual data), while relational property provides new information on the (abstract) relationship between the paired data.
Shen et al. 2017) Most these methods strive to learn a disentangles led representations
EXPERIMENT
In this section, we present experimental results from applying the proposed VRL method to set of relational learning tasks designed with the MNIST dataset (LeCun & Cortes 2010).
5.1 MNIST RELATIONAL LEARNING TASK
To setup relational learning task, paired dataset X (a b(0) [E was generated by the following steps: (1) the MNIST dataset { (x((), y (0) E where (() and y( are the digit images and their labels, was augmented with applying five evenly-spaced rotations to each of the image x(() to get { (x(+.)), ()) E T.T.je [1..5] } (examples of augmented images are shown in Fig. 3) (2) individual datapoint (a(9, b() of X was chosen to be randomly rotated images of (1) i.c.. = (a(+) = (().)), (()) = x6)0) I ~ u([I.T)), k, l ~ U([1..5)) }. Note that
Figure 3: Examples of a MNIST digit augmented with five evenly-spaced rotations (from left to right:
x0.1).0 :0, x(12), :72,, (i.33) 144 (f.4),216) (i.5) :288).
there are five uniquely defined rotational relationships between (a (() in X and since they are randomly selected, the rotational relationship (relational property) is decoupled from the absolute properties of (a()) b)). Here, X to assess VRL's capability of discovering the underlying

---

Under review as a conference paper at ICLR 2021
relational property (rotational relationships) irrespective of their absolute property, Additional MNIST relational learning tasks are introduces in appendix [.I
The MNIST relational learning experiment presented above may contrived at first glance but, upon deeper examination, represents a novel and unique problem setting that exemplifies a key relational learning laalllnnee-errrinnn learning methods (see Sec. for related work). We argue that any unsupervised method that can successfully solve the above problem (or any relational learning problem) must, minimum, simultaneously accomplish the following two goals:
decoupled relational property for existing unsupervised
1. An unsupervised learning mechanism that captures and preserves the data relationships (relational property), e-g. capturing and preserving the rotational relationship between (a()) (()) during learning.
2. An unsupervised learning mechanism that decouples absolute property from the learned data relationships, e.g., learning a rotational relationship between (a(4) b(0) that does depend the absolute property (digit representation, rotation, etc.) of individual a ((),
While most existing unsupervised methods well-equipped accomplish the first goal, argue that the second goal presents itself as a major challenge; more specifically, to the best our knowledge, most existing unsupervised methods focus on modeling all aspect (a b(²) and learn data relationships either jointly with the absolute properties of a b( or as their derivatives (including graph-based basph methods that edges to represent relationships). For such methodology, the learned data relationships necessarily entangle/couple with the absolute properties of the data, and therefore, the fundamental relational learning still unresolved. A key insight into why the proposed VRL framework is capable of overcoming this challenge is recognizing that VRL takes more targeted approach and learns data relationships stand-alone entity (represented by the latent variable z) that is designed, through the construction of VRL-PGM, to be independent of (however, we note that in VRL the learned relational property z is only independent of a but not b; discussion of this compromising fact and its implications is provided in appendix D.I.
relational property
5.2 IMPLEMENTATION
For these experiments, we adopted two-dimensional latent variable z E TR2 and let the prior po be the bivariate normal distribution. We let Pa (b/ a(f), (((.)) ) be a multivariate Bernoulli distribution whose probability parameters are computed from a given (1) and ((..)) with an autoencoder-like We let the approximated posterior qa 2 a (6) by()) be bivariate Gaussian distribution with a diagonal covariance N(z (t) (o(1)) where (1 and (i)
neural network fd (a(),, z (t.1).
are the output of a neural network (a br(1). For RPDA, we constructed a D that consists of image rotation augmentations: D = { (rot(a, r), r)) | r (6,3 }. Detailed experimental setup is described appendix
5.3 RESULTS
Relational discrimination. We trained VRL on X and used the approximated posterior go( b) infer the relational property of hold-out dataset. Figure (4] shows a scatter plot of the re- lational property inferred by VRL where we can see that the approximated posterior accurately cluster(disrriminate) data with the same(different) rotational relationship together(apart). Here, we compared VRL with variational autoencoder (VAE) (Kingma & Welling (014) and InfoGAN (Chen et af., 2016), both of which can learn data representations manner. In the application of VAE, we adopted 2-D latent space and use the encoder trained on to inferred the latent varible of a hold-out dataset; the resulting scatter plot is shown in Fig. 4b where we see that VAE failed to discriminate data based on their relational property (rotational relationship). Next, InfoGAN has demonstrated its ability to learn disentangled representations (represented by structured latent codes C1, C22 .... CL) through generative modelling. Although in- ferring latent codes for given data point a non-trivial task for InfoGAN, we can examine the learned latent representation by manipulating the latent codes and visually inspect the generated
in a completely unsupervised
7

---

Under review as a conference paper at ICLR 2021
(a) Relational properties inferred VRL
(b) Latent variable inferred by VAE
Figure 4: Scatter plots of the 2-D latent variable (showing only the mean μt) of hold-out datasets inferred by VRL in (a) and VAE in (b) (relationship labels: : V 72", + 144", 216". Q :88%).
random samples. We followed the examples in Chen al.) (2016) and modeled the latent codes with one categorical code e Cat( K" = 10, 0.1) (model discontinuous variation in data) and two continuous codes C2 C3
(-1, 1) (capture continuous variations in style). Figure shows examples of generated images from manipulating the latent codes; it is clear that none of the latent codes (or combination of them) distinctively capture the full range of relational property (rotational relationship). Figures 4b and iillstrate major challenge of using VAE and InfoGAN (and other related methods) for relational learning: these methods learn a single representation that encodes both relational and absolute property of X and it is difficult to dissect the relational property from the learned representation. Detailed experimental setup for both VAE and InfoGAN is described in appendix A
9
9
O
O
4
2
G
5
4
8
(a) Varying 02 from to InfoGAN
4
7
4
7
C
7
C
9
O
S
(b) Varying c from -2 to 2 on InfoGAN
Figure 5: Manipulating latent codes of InfoGAN on MNIST where each row represents random samples from varying continuous latent code 02 in (a) and in (b) while other latent codes and noise are fixed; different rows correspond to different categorical code C1.
Relational mapping. We evaluated VRL's learned likelihood function Po ( b a, by visualizing the
predicted images given a and z. We chose a from hold-out dataset and z from: (1) direct sampling
in the latent space; (2) relational property inferred from source datapoint (a,, b.). Figure shows
predicted images with z sampled from the latent space shown in Fig. 4a Figure 6b shows examples
of relational mappings from (c) to (r.c) by applying the relational property inferred from source datapoint (a., b." ). At first glance, the results in Fig. 6b resemble that of style-transfer, but they fundamentally different: in style-transfer, the image ((n)) is generated by applying the style of b(())
to the content of a (c), whereas VRL generates image (rie) by applying the relationship of (a,, b!") to the image a (c), It is evident from Fig. 6b that predicted images ((re)) do not share similar style to (rr) but rather the same rotational relationship w.r.t. (c) and a,-
8

---

Under review as a conference paper at ICLR 2021
2
3
4
5
6
7
8
9
O
3
2
3
4
6
8
2
2
8
4
5
6
7
8
9
O
3
4
8
2 (2)
R
9
3
6
6
7
4
0 6
3
2
9
6
2
2 3
5
9
< V
D
3
8
4
2
8
4
9
9
8
0
O
4
8
4
9
Z (5)
5
3
8
(a)
(h)
Figure 6: Examples of images predicted by VRL (a) images predicted from sampled latent variables
(sampling the centroid of each cluster in Fig. 4a
"O" + (1),, "v" + (2))) + (3) "x"
z(4), "Q" + z ()) and each image b (((e), r c 10, was predicted from an image (()) (shown in the top row) and pre-selected latent variable z using (rie) ~ Pe b a(*), 2(()) ); (b) examples of relational mappings top row images by applying relationships inferred from pairs of source images (a., b.")) (shown in the left-most column with a., b!.), ..... (b)) arranged from top to bottom) and each image ((ro), 1 Srrsh 5, 1 eees 10 was generated by b(ne) ~ po b a 2
where ((r) yo( z a,; bb"))
Additional experimental results and discussion are provided in appendix B and D.4, respectively.
6 DISCUSSION AND CONCLUSION
A core component of the proposed VRL method is approximating the intractable posterior with variational inference (VT) methods. There is vast literature on the subject of VI that we can leverage to further improve and extend VRL; for example, prior works have proposed flexible and complex approximated posterior distributions that we can use to learn a rich posterior approximations for characterizing the relational property (Rezende & Mohamed, 2015 Dinh et al. 2014). Another interesting idea to explore is learning a generative model for VRL: in this work, the primary learning objective is maximizing variational lower bound derived from VRL-PGM: however, with the advent of computationally efficient methods for learning a generative model, would be interesting to include another learning objective that directly models the data generating aspect of VRL-PGM
Goodfellow et al. 2014 Larsen et al. 2015).
The proposed method comes with both advantages and disadvantages: the main advantage of VRL lies in its relational learning capabilities; however, this may also be one of its disadvantages. More specifically, VRL can learn decoupled relational property even when it is coupled with the absolute property, i.e., VRL is oblivious to the coupling information between the two properties (an example with coupled relational property is provided in appendix I.I.I.. Nevertheless, such information may be of interest to the user, and in this regard, VRL provides only partial view of the data.
In conclusion, the proposed VRL method an efficient and effective unsupervised learning method for addressing the relational learning problem where our goal is to learn decoupled relational property. By dissecting the data information into decoupled relational and absolute property, we hope VRL can bring new insight into everyday data analysis and ultimately find applications for wide variety of problems.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and Koray Kavukcuoglu. Interaction networks for learning about objects, relations and physics. CORR, abs/1612.00222, 2016.
Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinícius Flores Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo. Adam Santoro, Ryan Faulkner, Çaglar Gülçehre, H. Francis Song, Andrew J. Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey R. Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matthew Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. CoRR, abs/ 806.01261, 2018.
N. Belhumeur, J. P. Hespanha, and D. J. Kriegman. Eigenfaces vs. fisherfaces: recognition using class specific linear projection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7):711-720 1997.
Irving Biederman. peaggnnttoo-by-oooooeeeett A theory of human image understanding. Psycho- logical Review, 94:115-147, 1987.
Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
Konstantinos Bousmalis, George Trigeorgis. Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. CoRR, abs/1608. 600.06019,, 2016.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pp. 10-21, Berlin, Germany, August 2016 Association for Computational Linguistics. doi: 10. 10.18653/v1/K166 1002.
Michael B. Chang, Tomer Ullman, Antonio Torralba, and Joshua B. Tenenbaum. A compositional object-based approach learning physical dynamics. CoRR, abs/1612.00341, 2016.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Picter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. CoRR, abs/1606.03657, 2016.
Emily L Denton and vighnesh Birodkar. Unsupervised learning of disentangled representations from video. In I. Guyon, U. Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 4414-4423, Curran Associates, Inc., 2017.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation, 2014.
Chris Donahue, Akshay Balsubramani, Julian J. McAuley, and Zachary C. Lipton. Semantically decomposing the latent spaces of generative adversarial networks. CoRR, abs/1705 701.775504774 2017.
Leonidas A. A. Doumas and John E. Hummel. Comparison and mapping facilitate relation discovery and predication. PLOS ONE, 8(6):188, 06 2013. doi: 10. 006.88.
Katerina Fragkiadaki, Pulkit Agrawal, Sergey Levine, and Jitendra Malik. Learning visual predictive models of physics for playing billiards, 2015.
Dedre Gentner. Language as cognitive tool kit: How language supports relational thought. American Psychologist, 71 (18) 11 2016. ISSN 0003-066X. doi: 10. 1037/amp0000082.
A.S. Georghiades, P.N. Belhumeur, and D.J. Kriegman. From few many: Illumination cone models for face recognition under variable lighting and pose. IEEE Trans. Pattern Anal. Mach. Intelligence, 23(6):643-660, 2001.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014.
10

---

Under review as a conference paper at ICLR 2021
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. bet:-vae: Learning basic visual concepts with constrained variational framework. In ICLR. 2017.
Felix Hill, Adam Santoro, David G. T. Barrett, Ari S. Morcos, and Timothy Lillicrap. Learn- ing to Make Analogies by Contrasting Abstract Relational Structure. arXiv e-prints, art. arXiv: 190:19900000200 January 2019.
Keith Holyoak. Analogy and relational reasoning. Keith J Holyoak and Robert Morrison (eds.), The Oxford Handbook of Thinking and Reasoning, chapter 13, pp. 234 259. Oxford University Press, New York, 2012.
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. CoRR, abs/ 804.04732, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), ICLR, 2014.
Anders Boesen Lindbo Larsen, Spren Kaae Sønderby, and Ole Winther. Autoencoding beyond pixels using learned similarity metric. CoRR, abs/1512.09300 2015.
Yann LeCun and Corinna Cortes. http:/yyaan.eeum.coo/eexdbiiii
MNIST
handwritten
digit database.
2010.
Michaël Mathieu, Junbo Jake Zhao, Pablo Sprechmann, Aditya Ramesh, and Yann LeCun. Disentan- gling factors variation in deep representations using adversarial training. CoRR, abs/161 111611.03300 103383, 2016.
Douglas L. Medin, Robert L. Goldstone, and Dedre Gentner. Respects for similarity. PSYCHOLOGI- CAL REVIEW, 100:254-278, 1993.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 26, pp. 3111-3119. Curran Associates, Inc., 2013a.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 746-751, Atlanta, Georgia, June 2013b. Association for Computational Linguistics.
Alan V. Oppenheim and Ronald W. Schafer. Discrete- Signal Processing. Prentice Hall Press, USA, 3rd edition, 2009. ISBN 0131988425.
John Paisley, David M. Blei, and Michael I. Jordan. Variational bayesian inference with stochastic search. In ICML, 2012.
Luis Perez and Jason Wang. The effectiveness of data augmentation in image classification using deep learning. CoRR, abs/1772.04421,, 2017.
David Raposo, Adam Santoro, David G. Barrett, Razvan Pascanu, Timothy P. Lillicrap, and Peter W. Battaglia. Discovering objects and their relations from entangled scene representations. CoRR, abs/ 702.00000,, 2017.
Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 1252-1260. Curran Associates, Inc., 2015.
In Cortes,
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows, 2015.
11

---

Under review as a conference paper at ICLR 2021
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi (eds.), Medical Image Computing and Computer-Assisted Intervention MICCAI 2015, pp. 234-241, Cham, 2015. Springer International Publishing. ISBN 978-3-319-24574-4.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. simple neural network module for relational reasoning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garett (eds.), Advances in Neural Information Processing Systems 30, pp. 4967 2017.
Curran Associates, Inc.,
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. Style transfer from non non-parallel text by cross- alignmett.... CoRR, abs/1705 hts//1005006555. 2017.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. How to train deep variational autoencoders and probabilistic ladder networks. CoRR, Abss/160..0282. 2016.
Le Song, Alex Smola, Arthur Gretton, and Karsten M. Borgwardt. A dependence maximization view of clustering. In IN ZOUBIN GHAHHAMMNII, EDITOR, PROCEEDINGS OF THE INTERNA- TIONAL CONFERENCE ON MACHINE LEARNING, pp. 815-822, 2007.
Joshua B. Tenenbaum and William T. Freeman. Separating style and content. In M. C. Mozer, M. I. Jordan, and T. Petsche (eds.), Advances in Neural Information Processing Systems 9, pp. 662-668.
MIT Press, 1997.
Sergey Tulyakov, Ming-Y Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. CoRR, abs/1707.04990 2017.
Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. CoRR, abs/ 2as/1802.10353. 2018.
Gary R VandenBos and American Psychological Association. Apa Dictionary of Psychology. Ameri- can Psychological Association, Washington, DC, 2nd edition, 2007.
Ruben Villegas. Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. CoRR, abs/ 701.7706....333 2017.
Jiajun Wu, Ilker Yildirim, Joseph Lim, Bill Freeman, and Josh Tenenbaum. Galileo: Perceiving physical object properties by integrating physics engine with deep learning. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 127-135. Curran Associates, Inc., 2015.
12

---

Under review as a conference paper at ICLR 2021
A EXPERIMENTAL SETUP
Recall that we adopted a two-dimensional latent variable z E R2 and let the prior po( z be the bivariate normal distribution. For binary valued data (e.g., MNIST dataset), we let the likelihood function pa{ b E B a+t), z(i()) of VRL be multivariate Bernoulli distribution whose probability parameters (6..)) are computed from given a and ((.)) with autoencoder-like neural network fg
fiee cene [a (a)) (t.))):
f8
renc 9
(1) Conv C3nv(3x3x8) Conv(3x3x32) - Conv(3x3x128) + FC((0) + h ( E
g [h R (1.) = FC Conv (3x3x(2)) Conv Conv (3x3x32) + Conv (3x3x8)
CCon((xxx1)
Sigmoid [0, 1]dim((8)
where Conv(-) is a strided (stride 2) convolutional layer and Conv (-) is a transposed convolutional layer. We used batch-normalization after most layers and leaky rectified linear units (with leaky rate 0.01) as nonlinear activation function. We let the approximated posterior qo( a(() b((() of VRL be a bivariate Gaussian distribution with a diagonal covariance (z: 11 ( where (+() and (i) are the output of a neural network fg (a (1) b(()));
f3 [a(()] b + C3nx(3x3x8) > Conv Conv (3x3x32) - Conv(3xxx128) + FC(() - (1) ,000
We sampled from the posterior (8.0) pz ( using ( (( a b((0) o)
(0 + (1) @ (f.1) where (ff.t) p(E) I) and set == We used image rotation augmentations: D= (rot(a. a, rot( rE [0, 360) } for constructing RPDA. In this case, the
(0 learning objective RNDA in Eq. can further be derived as (see next section for the derivation):
2
Z00
Σ
(1 + log((o, - (ngg))2 - (0)
2
1=1
+
Σ
(1) log (1.1) + (1 ((0) )
log(1 - -p/..1) (111))).
(8)
where (2 (3) fā(a" (1i) b(()), p (())= = = jdle (1om (a (a()). z(11).
(1.1) = ( + t((0) @ (i.)) ((i11) ~ ~(0,1)
(a b (10) = (rot(a()) (1) (()) rot(b(i) ,())), y(0 ~ ((0,3))))
Parameters 0 and & were jointly trained to maximize Eq. 8 using Adam optimizer (learning rate 0.0001, 91 0.9, 82 0.999) (Kingma & Ba 2014); minibatches of size M = 100 were used.
In our VAE implementation used the following network architecture for encoder, Enc(a(f) (()).
and decoder, Dec (z.)))
Enc :(a(() (a(()bbl) (00 + Conv(3x3x16) - C3nx(3x3x64) - (3n3(3x3x256) - FC(4)
((( ER? ERR
Dec :/2 (i.)) E ER² FC + Conv C (3x3x256) + (3x33(3)) + Conv (3x3x16)
+ CConv(1xx11)
Sigmoid
[0, 1 dim(1))
We used Adam optimizer with the same hyperparameter setup before.
For InfoGAN implementation, we followed the setup described Chen et (016) except that each datapoint consists a paired MNIST image (a()) b()).
13

---

Under review as a conference paper at ICLR 2021
A.1 DERIVATION OF TRAINING OBJECTIVE FUNCTION
Our derivation for Eq. ( largely follow the work of Kingma & Welling (2014); here for completeness outline the key steps. First, Eq. 44) in Section 3.2 can equivalently be expressed as:
( -DII (g (1) b ((0)
pa( z)
B (o log Pa (b()) ((01aa")) ,2)
DKL 10 (2)a( blu)
po(2)) py(e) log po (() a 22
(9)
where z - 5(E, (a,0b,b,,0 b(o), 6), p(e), and is some differentiable and invertible transformation. We can approximate (() in Eq. (9) with a Monte Carlo estimator Z())
E -DKL (zo(zal)))) blo) | po(z))
+ log Pe ((() a 2 (1.) )
1
(10)
where 200) = g(e(i,), ( b(i) の) and c(i.)) ~ p(e). Based on the RPDA functions D d(a, b; r } and its relation preserving assumption that 4s 22)a(), b()) (s(zaa")) bx())
for (6) b b(())= = d(a("), b(;; r), Vr (see Section 3.3 we can express (()) in Eq. (10) equivalently
EW
RPDA
DKI, [0l (6o(2|a") (0) b((1)) po(()) + Σ
log po (b() 2(10))
(11)
1=1
(11))) g(((1.)) (10) br(") ,)) e(id) ~p((),
where
(a (1) b(t)) d(a b(0)) p(0) U(R).
The first term in Eq. (1I) is the KL-divergence from po(z) to ga( z/aa" be computed analytically given that we assume pa(z) = NN(0,1) and qo (2) br())) = N2 (0) (0()))))
( br(,) which can
2 (+
-DKL go (z)aa()) bb(")) po po(2))
E6 i=1
-
(12)
The likelihood function po b (6)
(1) z (8.1) is defined as a multivariate Bernoulli distribution whose
probability parameters p (i.)) are computed from the neural network file (fa (a(*), 2(t.)) and we
have:
log po (6)
(8) z (0.0)
log (1.) + 162) log(1 Pk
(2.0)
(13)
By substituting Eq.
12 and 13 back in [11 and recall that (i.)) g(((1.)) (1) br( Φ) =
4 (1)
+ (+)
@ (a.1)
ppp (), D = (rot r), rot(b, re (0, (60) }. L=1, 1, we arrive
Eq.
B ADDITIONAL EXPERIMENTAL RESULTS
B.1 ADDITIONAL. MNIST RELATIONAL. LEARNING EXPERIMENTS
Here, we provide additional experimental results based on the paired MNIST dataset constructed in Section
B. MNIST EXAMPLE WITH COUPLED RELATIONAL PROPERTY
First, further test the robustness of the proposed method, we considered scenarios where the underlying relational property is coupled with the absolute property. An example dataset X was constructed with the rotational relationship between each of the datapoint completely determined U((I.T), k. u(11.5)),
by its digit label: X² (+) x(ik)_b x(1.+1))
14

---

Under review as a conference paper at ICLR 2021
(k + [20) mod 5). In this case, it is possible infer the relational property solely based on the image representation of the digit (absolute property of ) for example, a [0'' '1' 0 (read: if a is recognized either digit 0 or 1, the rotational relationship between (a b (() is a [22 '3"] + 72". ( E ('4', '5'] 144", a E ['6', '7"] 216", a E ['8' '9'] - 288".
The question then arise: is VRL capable learning a decoupled relational property even when is coupled with the absolute property? To test this idea, trained VRL (with the same setup that described in appendix A) on X" but validated on a hold-out dataset with a decoupled relational property (much like how X was constructed). The resulting scatter plot is shown in Fig. 7 where we can see that the approximated posterior 4s z a, b) accurately cluster(discriminat)) data with the same(different) rotational relationship together(apart). This result shows that VRL was indeed capable of learning decoupled relational property irrespective of the digit representation of ( this were not the case, we would expect to see scatter plot with heavily overlapped relationship labels since the validation dataset constructed with random rotational relationships. Examples
3-
2-
-3
21
Figure 7: Scatter plot of the relational property (showing only the mean μ4) of hold-out dataset inferred by a VRL model that was trained X (relationship labels: O 09, V 72%, + 144 x:216", 288%).
of images predicted by the learned likelihood function Po b a, z are shown in Fig. where Fig. 8a shows predicted images based on direct sampling in the latent space (shown in Fig and
Fig.
8b
shows examples of relational mappings. Figures Sa and [b further corroborate our claim that
3
4
5
6 7
8
7
3
2
3
4
9
9
Q
2
2
3
4
5
6
7
2
7
O
3
P
41
7
9
Q
2 (2)
3
O
M
C6
e
L
3
z
3
8
0
5
9
3
6
4
0
Z
6
b
8
R
4
5
9
Σ
b
O
z
W
3
(a)
(b)
Figure Examples of images predicted by a VRL model that was trained on 22: (a) images predicted from sampled latent variables (sampling the centroid of each cluster in Fig. 7 "!') " (2),* "x" "x"++((1) "Q" z (5) (b) examples of relational mappings of top row images by applying relationships inferred from pairs source images (a,, b!")) shown in the left-most column with a,, b b. arranged from top to bottom.
(5)
15

---

Under review as a conference paper at ICLR 2021
VRL is capable of learning decoupled relational property and generalizing it to unseen data, e.g.. VRL trained on X² learned to rotate any digit by any amount despite not having seen most of the digit-rotation pairs during training. Note that this experiment is example of deterministic mapping discussed in Section 3.3 nevertheless VRL was able to utilize all paths in Fig. 2bjeffectively to learn about the data.
B.1.2 MNIST EXAMPLE WITH MULTIPLE RELATIONAL PROPERTY
Next, setup more complex relational learning task that includes both rotational and resizing relationships between (a()) b b()): each MNIST image x()) was augmented with five evenly- rotations and three different resizing transformations to get { (x(+.kk), ( [ [1..5]. & [1..3) }. Examples of augmented images are shown in Fig. where top, middle, and bottom row images are resized by a factor of x x0.66, x1, and x x1.5, respectively. We considered relational
3
3
3
3
3
3
Figure 9: Examples of MNIST digit augmented with rotational and resizing transformations (from left to right: ((.11k); 00, x6..2.0) 722, x(f.3.k) 1444, (ff.4,) 2116,, x11,5,8) 288%; from top to bottom: (f1..11 x0.66, (f.j.2) x1, x(1.5))))11.))
learning task where each datapoint (a b()) has a decoupled rotational and/or resizing relational property; more specifically, we constructed dataset X in the following way:
{ (a (i) = xlide.) (6) x(id.))) j ~ U([I.T]), k, ~ U([1..)), m U(11 2)), ~ (VV)
[1,2]] if m =
where
v= [2,3], if =
Note that in this case, by()) is either the same size as a or is x1.5 larger than a (0) and there are total of 10 different relationships between (a b(9) in X (combinations of 5 rotational and resizing transformations). We trained VRL on X with the same model setup as that described in appendix A and used the trained model infer the relational property of hold-out dataset. The inference result is shown in Fig. 10 where we can see that the approximated posterior 2s a, b accurately cluster(discrmminae data with the same(different) relationship together(apart), Examples of images predicted by the learned likelihood function Po b
are shown in Fig. I1 where Fig. IIa shows predicted images based on direct sampling in the latent space (shown in Fig. IO). and Fig, shows examples of relational mappings. These results are consistent with the findings presented and discussed Section 5.3
B.1.3 MNIST EXAMPLE WITH CONTINUOUS RELATIONAL PROPERTY
Finally, we present an example with a continuous relational property. Based on the MNIST dataset (x()) (0) iE [I..T) we constructed paired dataset X in the following way:
x = (= (a x0, (0) = rx) (1)) j ~ U((I.T)) 10 ~ U( 360)) }.
where rot (x(), p()) rotates the image x by (+) E 360) degrees. this case, (6) a random rotation (0) and there is a continuous (and decoupled) rotational relationship between (a) b()) in X We trained VRL on X with the same model setup that described in appendix A and used the trained model to infer the relational property of a hold-out dataset. scatter plot of the relational property infered by the approximated posterior q images predicted by the learned likelihood function Po b a, z are shown in Fig. 13 where Fig, 13a shows predicted images based direct sampling in the latent space (denoted by markers "x" in
b is shown in Fig. 12, and examples of
16

---

Under review as a conference paper at ICLR 2021
3
2-
22
0-
-3
Z1
Figure 10: Scatter plot of the relational property (showing only the mean μt) of hold-out dataset inferred by VRL model that was trained on X (relationship labels: Obblue) :0", 72 144", 144",
x :2266,, 288 = x x15,, D 72", x 5, <I 11:449, x1.5, A 2161,, x1.5, O(cyan) 288 x x1.)).
3
7
8
9
O
3
0
6
3
2 11)
/
4
3
8
5
6
7
9
3
4
6
6
2
7
0
m
7
5
4
4
<
9
0
2 3
4
3
9
7
9
O
Z
9
4
4
6
Z
3
5
Z
Z
M
0
8
Z
z 99
(10)
(a)
(b)
Figure 11: Examples of images predicted by VRL model that trained on X3. (a) images
predicted from sampled latent variables (sampling the centroid of each cluster in Fig. TO "O"(blue)
(1), "V" (2) " "+" (3)) "x" - (4),, "Q" ""0"+2(0) (5)) "O" (6),, "D> + (7) "0"""""" " (8) "A" z (1));; (b) examples of relational mappings of top row images by applying relationships inferred from pairs of source images (a,, b)r shown in the left-most column with (1) b (10) arranged from top to bottom.
( "cyynn)))
a. b
Fig. 12 and Fig. 13b shows examples of relational mappings. From Fig. 12 and 13 we can see that VRL learned decoupled relational property that encodes continuous data (rotational) relationship: however, there is a small region in Fig. 12 with overlapping relational property that leads to an ambiguous interpretation of the rotational relationship (120* vs. 240"). This ambiguity is likely caused by compressing the continuous data (rotational) relationship down to a two-dimensional latent space, z E IR?, and motivates us to adopt higher-dimensional latent space, e.g., z E R3 Figure 14 shows inference result from repeating the previous experiment but with setting 3 see that VRL learned a dire-dimenssonall relational property that unambiguously represents the underlying continuous data (rotational) relationship.
17

---

Under review as a conference paper at ICLR 2021
360
2
300
1
240
42
180
120
60
-3
-2
2
Z
Figure 12: Scatter plot of the relational property (showing only the mean μı) of a hold-out dataset inferred by VRL model that trained on X*; each point is color-coded (best viewed in color) by the degrees of rotation between the corresponding datapoint (markers "x" denote sampled latent varibles z (1), z5)) used for image prediction in Fig. 13ah.
2
3
4
5
8
3
2
4
5
6
4
2 (
2
3
4
5
6
8
3
2
E
9
z 2
3
3
5
6
3
8
0
3
9
3
5
G
(
5
9
a
8
6
3
5
5
6
2 4
3
5
5
8
0
6
5
9
(6)
2
6
9
(a)
(b)
Figure 13: Examples of images predicted by VRL model that was trained on X (a) images
predicted from sampled latent variables (denoted by markers "x" in Fig. 12]: (b) examples of relational mappings top row images by applying relationships inferred from pairs of source images
(a,, b(²)) shown in the left-most column with a,, (1) ..... b. arranged from top to bottom.
B.2
YALE FACE RELATIONAT. LEARNING EXPERIMENTS
B.2.1 YALE FACE EXAMPLE WITH LEARNING EMOTION CHANGE
Here present another relational learning example based the Yale face dataset (Belhumeur et al.) 1997). The Yale face dataset consists of 15 subjects, each with facial expressions and lighting configurations. We extracted three facial expressions (happy, surprised, sad) of each subject to form { ((.) center-cropped, resized to 64 x 64, and normalized pixel values to be within [0, 5j. We constructed
E [1..3] J. Examples of face images are shown in Fig. 15a where
dataset F. where each datapoint (a(0) b()) represents a subject with different facial expressions (emotions):
X (a (i ( (())) 1j (((1..15), & &~ U((..))), 1 1~ 4(1.33) })
Our initial intention was to apply VRL F. to learn about the "emotional change" between (a()) b()) irrespective of the subject; however, because X is extremely limited dataset that
18

---

Under review as a conference paper at ICLR 2021
300
180
Figure 14: Scatter plot of the three-dimensional relational property (showing only the mean (4) of hold-out dataset inferred by a VRL model (with z E R2) that was trained on X each plot shows different vantage point of the 3D scateer plot, and each point is color-coded (best viewed in color) by the degrees of rotation between the corresponding datapoint.
89
-1.5
-1.0
-0.5
Z1
(a)
(b)
Figure 15: Learning emotional changes among the Yale face dataset: (a) examples of subjects with different facial expressions: happy, surprised, and sad; (b) scatter plot of the relational property (showing only the mean 14) inferred by the approximated posterior (relationship labels: V: "happy- sad", O: "happy-surprised", +: "surprised-sad").
consists only 45 different images (15 subjects, each with 3 facial expressions), we settled for a less ambitious goal of learning undirected emotional change, i.e., "happy->sad" = "sad- -happ-hhpp" and simply denoted as "happy-sad", In this case, there are three different emotional changes between (a()) (() in X "happy-sad", Shappy-surprised) "surprised-sade training, adopted two-dimensional latent variable
To setup a VRL model for I2² and let the prior pa(z) be the
bivariate normal distribution. Since this is real-valued dataset, we let pa( b(() (+)) ((.)) be
a multivariate Gaussian distribution with a fixed diagonal covariance (b pb o2) where
(i.))
(6.1) is computed from given a and (+.)) with autoencoder-like neural network 0.1 and 1
19

---

Under review as a conference paper at ICLR 2021
g (Jgme (al 2(:1)::
S a (0) + CCnv(333x4) - Conv(3x3x16) - Conv(3x3x64) - FC(50) - (0) E E
foee [h ER=0 , ((.)
FC -
-
+
Conv (3x3x63)) - Conv C (3x3x16) Conv (3x3x44
+ Conv(lxlxI) + Pb
(1.1)
and we have:
lbb
(1.1) Pb
12
Po( ( a (1) (6.1))
+ const.
(14)
2
0.01
b be a bivariate Gaussian distribution with a diagonal covariance (zz ( (a (1 where (0()) and o(1) are the output of a neural network
Again, we let the approximated posterior go( (6) b
f brlt)):
f3 (a(()) b Conv(333x4) - Conv(3x3x16) - Conv(3x3x64) - FC(() - [1e()) (0)
Next, based on the premise that we are only interested in learning an undirected emotional change, we have po a, b) po(2 z b, and this prompted construct RPDA functions D with random image rotation and swapping operations: D= swap (rot(a, aa, r), rot(b, r)) r [0, 360) } where
swap(a, b) [ b), p p= 0.5
(b, p 0.5
Finally, we combine the above settings with (1..) = n + a (t) @ clt,) = M( 1). I-11, and substituting Eq. (12) and (14) back in (IT) to derive the following lower bound estimator:
CRPAA
0.01
1 + log((o, -
-
b -1b
(1.1)
const. (15)
(1()) ( fg(a/) (1) b(())),
(a.1) pdac ( (ali)), (sam((al)) 2(:11).
where
Hbb
(611) 1 + +o(1) eli)), (li1)) N(0.1).
(a (,)))) swap (rop(rot(a,, rot(b,
(() 3(0,360))
The rest of the training setup remains the same as that described in appendix A We trained VRL on F and then used the approximated posterior go a, b)) to infer the relational property of F. The inference result is shown in Fig. [5bb] where we can see that VRL learned a relational property that accurately differentiates emotional changes irrespective of the subject.
B.2.2 YALE FACE EXAMPLE WITH LEARNING ILLUMINATION CONDITION CHANGE
Next, we present an experiment that learns relationship on "illumination condition changes" among the Extended Yale Face Database B (Georghiades et al.] 2001). The Extended Yale Face Database contains 16128 images of 28 human subjects under poses and 64 illumination conditions. We extracted four illumination conditions (source of illumination: left, front, right, top) of each subject
to form { (t.i i [1..28].5 E [1..4] }. Examples of face images are shown in Fig. 16a where we center-cropped, resized to x 64, and normalized pixel values to be within [0, 5). We constructed dataset F where each datapoint (a(*), ((()) represents subject with different illumination conditions (lightings):
X Fi ==== (a = x(M)) b x())) 1 ((1.228)), l ~ U((1.4)), 1~ 4((1.4) (b) }.
Like the "learning undirected emotional changes" example presented in appendix B.2.1 we apply VRL to F to learn about the "undirected illumination changes" " between (a( ( irrespective of the subject, i.e., "left-sright" "right and simply denoted as "left-right", In this case, there are six different illumination changes between (a (() in Fi; ("LL--") T"), feettfrront("L-F"") ("LFF", "left-top" "'"-T"", "front-right' right"("F-R"" ("F-R"), ""ghh-ttpp("RRT"") We trained VRL on F with the exact same model and training setup used in appendix B.2.1 and then used the
("L-R"), "front-top"("FF
20

---

Under review as a conference paper at ICLR 2021
0.5
0.0
2.0
(a)
(b)
Figure 16: Learning illumination condition changes among the Yale face dataset: (a) examples of subjects with different illumination condition (source of illumination): left, front, right, and top; (b) scatter plot of the relational property (showing only the mean μ1) inferred by the approximated posterior (relationship labels: V: "left-right", Q: "front-top", O: "left-front". +: "left-top", x: "front-right", O: "right-top").
approximated posterior go( b) to infer the relational property of F The inference result is shown in Fig. 16b where we can see that VRL correctly identifies four relationships ("L-T", (-L-T".-R-T "R-T", "F-R". "L-F" each represented by distinct cluster in Fig. T6b) but collapses "F-T" and "L-R" into a single indistinguishable cluster. At first glance, this counterintuitive result collat-oll of "F-T" and "I.-R"
indicate that VRI. and "I.-R"; however, there an elegant and logical explanation that justifies this unexpected result. Upon closer examination, we argue that it is indeed possible to combine the relationships "F-T", "L-R" without loss of information. In fact, we can give the combined relationship meaningful name- "Oppostt direction illumination condition" or "Oppo." for short (in this interpretation more intuitive to rewrite "front" as "down""). With the newly formed compressed set of relationships R. = ("L-T", "R-T". "F-R". "L-F". "Oppo."), it easy to see that R. a valid of relationships that fully and unambiguously characterizes the data (thus no loss of information) since: 1. any (a()),b (() can be characterized by one and only one relationship in Re; 2. for any given a each compatible relationship in R. applied to ! leads to a unique (()) And while we lost the identifiability "F-T" and "L-R", combining them into "Oppo." is consistent with the relational learning goal-"Oppe represents relative relationship that does not depend on the illumination condition (absolute property) of each individual image. is not always guaranteed to learn compact set of decoupled relationahips, and we will leave the investigation of this idea to a future work. Finally, we note that although the initial expection of learning complete set of relationships is certainly reasonable and intuitive, it is quite surprising and unexpected that VRL is capable of discovering- a completely unsupervised manner a non-obvious set of relationships that is equally valid, and yet more compact.
not able learn meaningful representations for "F-T"
stress that the proposed VRL method
To summarize the Yale face experiments presented in appendix B.2 we make the following two observations. First, when comparing results with existing unsupervised leanring methods face images (Song et al. 2007), we can see that existing methods cluster face images by its absolute property (e.g., subject identity, individual emotion, individual illumination condition, etc.) while the proposed VRL method clusters images by relationships (c.g., emotional change, illumination condition change). Second, is worth noting that VRL does not learn the facial expression or illumination condition for each of (a b()) independently and then classify them based on their difference, but instead directly learns the emotional/lighting relationship between (a (()).
21

---

Under review as a conference paper at ICLR 2021
C ABLATION STUDY ON RPDA
In this work, we introduced cllatin-ppeeerrving data augmentation (RPDA) as a practical data augmentation strategy for overcoming the information-shorteut problem-an unique overfitting problem introduced by VRL (see Section 3.3) In order to evaluate the contribution of RPDA, we
performed ablation study on the paired MNIST dataset X constructed see Section 5.1 First, recall that in Section [5.2 we constructed RPDA functions D with image rotation augmentations and optimized the following lower bound estimator (cf. Eq. (7):
RPDA
L
Σ
log pa (0)
( ,2 (6.0) + log (zz (1.)) log (z(26))))) a ()))) (16)
(i,) g(( a()))) br 2) ((id) ppE))
where
b)))) d(a(") b(0, b(1,)(0)) (() U()..
In the first ablation study, we experimented with removing RPDA from the VRL training, which amounts optimizing with the original lower bound estimator [ in Eq. (6) With the rest of the training setup remained unchanged (see appendix(A) [A), we trained VRL on X without RPDA and used the trained model to infer the relational property of hold-out dataset; the inference result is shown in Fig. 17a In the second ablation study, we repeated the VRL training, but instead of following the proposed RPDA strategy, we applied RPDA functions D in a conventional data augmentation routine. More specifically, we optimized the lower bound ((0) in Eq. (6) over a minibatch of data augmented with D, M = (a (9) b(()) E [1..M]}
}. which leads to optimizing with the following lower
bound estimator:
200 DA
log po
(0) ( ) log po (z log 90 (z((1)) a(((), b(())
(17)
L
I=1
z (1l) = g(e(t.t) a (1) br(1) ,), ((il)) ~ pp()),
where
(a (+) b(()) = ((a b(0 7(0) U(R).
With the rest of the training setup remained unchanged (see appendix A). we trained VRL on with maximizing Lox and used the trained model to infer the relational property of a hold-out dataset; the inference result shown in Fig. 17b Comparing the results from both ablation studies, shown in
9
8
-1-
-2-
dP
-2
22
0
Z1
zi
(a)
(b)
Figure 17: Scatter plots of the relational properties (showing only the (1) generated from VRL
ablation studies (relationship labels: O: 0%, V 72", + 144", x 216", Q 288%): (a) relational
property inferred by VRL model that was trained without RPDA; (b) relational property inferred by a VRL model that was trained with applying RPDA functions D in conventional data augmentation routine.
Fig. 17 with inference result from VRL training with RPDA, shown in Fig. 4a
make the
22

---

Under review as a conference paper at ICLR 2021
following observations: first, RPDA is critical component necessary for VRL to learn meaningful and decoupled relational property, especially when flexible function approximations such as deep neural networks are used; second, by looking at the difference between L(i) and ZRPDA (observe that I applies RPDA functions D to every term in Eq. (77), whereas _ applies D only to selected terms in Eq. (16) according to the RPDA strategy), we can draw the conclusion that the improvements brought by RPDA (and its key innovation) comes not from what data augmentation functions are applied but how they are applied.
D ADDITIONAL REMARKS
D.1 REMARKS ON VRL-PGM
The central idea of the proposed VRL method is to encapsulate the relational learning problem with a probabilistic graphical Vodde--VRLLPGM--- as performing inference and learning in the graphical model. Here we discuss aspects of the original relational learning problem (see Section 2 that differ from the proposed VRL-PGM (see Section 3.1) First, the original problem specifies that the relational property be decoupled from both a's and b's absolute properties; however, the latent variable z that is used to represent relational property in VRL-PGM is only independent of a but not b. Second, we note that the original problem is inherently undirected with no cause-effect relationship between a and b, whereas VRL-PGM is based a directed acyclic graph (DAG) that artificially introduces conditional dependency between and b. However, we argue that the application of VRL does not require the true conditional dependency between (a, b) be known in advance only that it is maintained consistently throughout learning and inference, i.e., VRL can be applied in the same way to learn about the relational property between (b, a), where swap and b. The above-mentioned two discrepencies represent the compromises with adopting VRI-PGM in exchage for a riorous and tractable method for learning a decoupled relational property. We can futher view VRL's optimization and ceermaation-ssoorte (introduced in Section 3 as the consequence of these compromises: deterministic- mitppppnnsiiimmapinn can viewd caused by the causal relationship VRL-PGM introduced between and b. and commaatioo-soorttu can be viewed as caused by the causal relationship VRL-PGM introduced between z and b.
then formulate various relational processing tasks
made
daallengee-ddeemmminstiii mapping
D.2 REMARKS ON INDEPENDENCE BETWEEN Z AND a
Recall that the assumption of independence between z and is central to the VRL learning of decoupled relational property, and in this work we rely on the construction of VRL-PGM to support such assumption; however, the learning objective (Eq. 4 derived from VRL-PGM does not explicitly force to be independent of a (nor penalize learning dependent z). Here, we explain how optimizing Eq. 4 affects the learning of independent z and a. The main learning objective of VRL is described in Eq. where we can further dissect its various terms and observe that the learning is heavily guided by maximizing the likelihood term log Pa( b z) since this is the only term that is constrained by the data while all other terms in Eq. 4 can be viewed regularization on the unobserved Next, when maximizing log Po (b0 (+) 22) we are only learning to predict (60) from ( and z (0) and since ( is already been conditioned on, there is no incentive for ( to learn redundant (dependent) information from a (1), This effect is further "encouraged" when the learning objective Eq. also includes additional regularization terms on z for learning a compact representation that "squeeze-out out" any redundant information that does not help with predicting b((), In the information flow diagram shown in Fig. 2a one may argue that since a also propagate information through the latent variable it may introduce dependency between and z; however, the information propagated from a through ( is mainly used to maximize the likelihood term for predicting ()) and since there is already a direct propagation path from (1) to ((0 (as part of the likelihood function po b ( (a(() z )) there is, again through regularization on z, no incentive for z to carry redundant information from a (only decoupled relationship information derived from and b). In short, while there are explicit penalties on learning dependent z and a in VRL's learning objective, the independence is naturally encouraged through the interplay between the different terms in Eq. 4 This effect is also corroborated by our experimental results where we can see that VRL can indeed learn a decoupled (independent) relational property z through optimizing Eq.
23

---

Under review as a conference paper at ICLR 2021
Ideally, the learning of independent and can be achieved through VRL's learning mechanism discussed above; however, in practice there may be numerous reasons that could cause this to fail, such as insufficient training data, failure to reach the global optimum, non-identifiability of the model, etc. Therefore, for pragmatic reasons, it may be desirable to explicitly safeguard against introducing dependency between z and Here, we propose straightforward extension to VRL. for achieving this goal: we can append any non non-positive function that measures the dependency between a and z with maximum attained when they are independent to Eq. 4 without invalidating the lower bound. For example, we can append the negative mutual information between z and a, I (z,,a) a) = ( (z.a)
po z ppo a )). to Eq. (4) to obtain:
C
L (0 log Po (0) a (1)) ) + log po z)) log qsl z/2a b(0) (z, a). (18)
And since [zz a) 2 0 and [ a) = 0 if and only if z and a are independent, the addition of (z, a) in Eq. (18) not only does not invalidate the original bound in Eq. ( but also does not sacrifice the quality of the lower bound (z and a are independent in VRI-PGGM). We will leave the
investigation of this idea to a future work.
D.3 REMARKS ON RPDA
Here would like to comment the practical applicability of the proposed RPDA stratey. More specifically, we would like to convey the idea that in many practical problem settings, the RPDA functions D can be designed without any knowledge of the underlying relational property. For example, as we have explained in Sec. 3.3 in many computer vision applications, rotation invariant is a desirable property for the learned model; for example. in spectral imaging applications, oftentimes the orientation of the images are not preserved or not enforced (only that they are consistent between the same paired images) (Ronneberger et al., 2015). In such problem setting, we can safely use image rotation function for constructing D. Another example may be: for a discrete time-series data a/t), b(t] that represent the input and output of linear time-invariant (LTT) system (commonly assumed in signal processing and control theory (Oppenheim & Schafer 2009)), and we want to learn a relational property that characterize the system's 's impulse response. We have ab/t - 7] = aalt- 7]. Va E R, T Z, and we can construct D with d(a(t), b[t]: a, F) (aa(t - 7], ab/t r]] a, T E = R x Z. In all of the above examples, the construction of RPDA functions D reflects our prior knowledge and belief of the underlying system and not based on the data relationships; therefore, in many instances, RPDA can be designed without any knowledge of the underlying relational property. However, we would also note that RPDA is not central to the theory of the proposed method-VRL can be applied without RPDA-but rather a practical data augmentation strategy for addressing unique optimization challenge (information-shorteee of our experiments, we find RPDA to be effective and crucial in overcoming the information-shorten problem (as illustrated in appendix (.. But just like any data augmentation, this is problem dependent and we advocate to start without RPDA and only apply it when suspecting information-shorten occurs.
of VRL learning. In all
D.4 REMARKS ON EXPERIMENTAL RESULTS
Here we would like to give overall summary and make general remarks the experimental results we have presented in this work (included in Sec. Slad appendix B First, we proposed four MNIST experiments that represent a diverse set of relational learning problems: decoupled relational property (Sec. 5. coupled relational property (appendix T. multiple relational properties (appendix(B.I) 1.1.)). and continuous relational property (appendix [...)). Although these experiments are introduced with well-controlled ground-truth relationships (so that we can easily validate and interpret the results), the application of the proposed VRL method is completely unsupervised without any prior knowledge of the underlying relationships. Furthermore, we deliberately design and successfully solve all four MNIST tasks using the exact same model and training setup despite each experiment represents very different relational learning scenario (discrete vs. continuous, coupled vs. decoupled). In addition, we presented two Yale face tasks with high-level perception relationships: change of emotions (appendix B.2.1). and change of illumination conditions (appendix B.2.2). Again, the application of VRL to the two Yale face tasks is completely unsupervised and we deliberately design and successfully solve these two tasks using the exact same model and training setup.
24

---

Under review as a conference paper at ICLR 2021
When comparing our model and training setups between the MNIST and Yale face experiments, their differences can all be attributed to the need for accommodating different data types, c.g., increasing nework size for larger face images (vs. smaller MNIST images), modifying learning objective to adapt real-valued Yale dataset (vs. binay valued MNIST dataset), and not for accomodating different relationships. Taking this into consideration, problems with the exact same principled method despite each class of problems represents a very different kind of relationships (the relationships in MNIST are geometric whereas the relationahips in Yale are high-level perception, e.g., sentiment, external environmental factors). We believe our results further demonstrates that the proposed VRL method is robust, stable, and generalizable to many different relationships.
results shows that have solved both class of
25


Reviews:
[{"review_id": "r-VlSAt_wWy", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the author proposed a variational framework for relational learning that decouples relational property and absolute property among objects based on a pre-defined probabilistic graphical model (PGM). The author also proposed the so called relation-preserving data augmentation (RPDA) strategy to address the challenges for the resulting optimization. Overall, the paper is well written and easy to follow. Below are some of my concerns.\n\n1. It seems that RPDA is crucial for training. However, the relation preserving function D is often difficult to find for real data where the existing relation patterns are often unknown.\n\n2. No ablation study is reported regarding the effect of RPDA.\n\n3. Experiments seem to be too simple. It might be hard to find appropriate application cases though.\n\n4. Can the author clarify the relation between the proposed PGM based relation learning framework and causal learning?", "extra_scores": {}, "extra_text": {}}, {"review_id": "XqrK4m4QH2", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Relational learning is an important capability exhibited by humans of learning relations between objects.  This work considers a fairly general setup for relational learning and addresses learning of the relation through a method based on variational inference.\n\nI am not familiar with the VAE literature; having said that, the method seems novel.  The problem setup is so general that it is unlikely to be novel, but the main technical contribution of the paper appears to be the derivation of the objective (4).  Of course, given that the objective is a lower bound of the probability we want to optimize, and since no theory is provided on the quality of the approximation, it is important to have strong demonstrations of the method in which it is shown to have unique advantages compared to existing approaches.  Unfortunately, the paper falls short in this regard, as I will go into more detail.\n\nThe equation (4) does seem plausible and it is rather elegant.  Note that (4) can also be seen as likelihood + regularization on z + entropy of z, hence it encourages latent variables z that are not too large according to the regularization, while the entropy term prevents z from collapsing to the mode.  This makes sense, but at the same time it is difficult to see how independence between z and a would be enforced using this objective function.\n\nThe other technical contribution is the identification of optimization issues via the \"information shortcut\" and \"deterministic mapping\" issues.  This is very insightful.  The idea of RPDA is interesting, and in many cases it could be applied at least via data-augmentation.\n\nThe main demonstration is based on learning rotation angles in MNIST.  The relationship of \"A is 30 degrees rotated compared to B\" does qualify as a relation, albeit one that seems fairly artificial.  (EDIT: since discussion, I retract the following in brackets).  [More worrisome is that the example used, if I understand correctly, does not follow the causal model adopted by the authors.   The model assumes that a and z cause b.  However, the example has b as the MNIST label.  This is causally incorrect because for handwritten digits, the label is the cause of the image, not vice-versa.  Presumably the causal assumptions should be correct for the model to work well?  It would be helpful if the authors could comment on this.]\n\nThe Yale face dataset in the supplement seems a better fit to the causal assumptions.  \n\nHowever neither example is satisfactory in showing a unique benefit of this approach that cannot be already obtained by domain-specific methods, such as spatial transformer networks in the case of learning image rotations/scaling (https://papers.nips.cc/paper/5854-spatial-transformer-networks.pdf) and emotion learning ML methods for facial images (https://www.sciencedirect.com/science/article/pii/S1877050917327679).  Perhaps the authors could propose an application that would showcase the advantages of the approach, even if it is currently infeasible to obtain results?\n\nIf we contrast the relational learning approach to the existing body of work on learning disentangled representations, the main difference is that in existing representation learning work, one learns the latent variable _z_ from _a_ only.  But here, _z_ can be learned from both _a_ and _b_ .  This is obviously an important difference, but one could reduce the problem of learning _z_ from _a_, _b_ to methods that learn _z1_ from _a_ and _z2_ from _b_ separately.  For example, in learning the rotation between _a_ and _b_ (both MNIST digits), one could reduce it to learning the orientation of _a_, learning the orientation of _b_ separately, and then subtracting the orientations.  Of course, this seems to be a property of the examples used rather than a general fact.  If one considers more sophisticated relational learning problems such as properties and objects, then it may no longer be reducible.  Yet this again indicates that the examples used in the paper fail to show the uniqueness of the proposed approach.\n\n### Pros:\n * interesting problem with applications to psychology\n * insightful analysis of the optimization involved and remedies for two pitfalls\n\n### Cons:\n * toy examples inadequate to show novelty and generalizability of the method\n * not obvious to me how the objective function encourages independence of _z_ and _a_.\n * do we have any confidence that this method would work in a more complicated problem?\n\n### Recommendation:\nThe paper starts an interesting line of development but still seems to need more work in either the theory, the sophistication of demonstrations, or both, before one could make a convincing case that it adds something beyond similar approaches in the literature.  Hence I recommend rejection of this paper.  The authors are encouraged to look into adding more sophisticated and unique examples, and further analyzing their method.\n\n### Modifications since discussion\n * retracted objection about inappropriateness of MNIST example\n * raised score from 5 to 6", "extra_scores": {}, "extra_text": {}}, {"review_id": "8fWV9s4-A1", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a model to infer the relationship between multiple instances in a dataset by inferring a latent variable. The authors accomplish this by defining an optimization problem that optimizes the ELBO of the proposed graphical model. The paper presents a nice solution to some of the identification issues that can arise when inferring the latent variable, in particular the so called “information shortcut” when the model overfits to only learning the “absolute” property of the dataset, rather than inferring the shared latent traits. \n\nI think that the formulation that the authors present and the proposed  optimization model is quite interesting, however I have additional questions / concerns:\n\nIt’s not clear exactly what the authors mean by a relationship.\nWhy is there a latent variable causing b but not a? Why are we assuming that a causes b? \nIt’s unclear to me why we refer to the relational property as ‘z’. According to figure 1 z has no effect on a, so it doesn’t seem like it would describe the relationship between a and b? Wouldn’t theta be serving that purpose?\n\nI’m also confused about the difference between what is proposed here and a latent factor model. It would appear that the authors are proposing a model to integrate a latent factor model into a generative net, which is interesting, but does not come through in the text as it currently reads. It would seem that in order to interpret z as a relational variable we should be able to extract some kind of meaning from it? Also, if there are multiple relationships , say “animal”, “rotation”, and “saturation”, should we expect to be able to disentangle these concepts with the proposed model?\n\nI found the experiments to be a bit underwhelming. It is unclear how the authors decided on the architecture, hyperparameters, and number of latent variables for the proposed model. It would seem that the model would be quite sensitive to these. In addition, it seems that some of these evaluations would benefit from comparison to more traditional methods. For example, the faces example appears in “A Dependence Maximization View of Clustering”, within a very similar context.  \n\nOverall, I think this is an interesting idea, but I would like to see the paper a bit more refined before recommending acceptance. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0SA9ofX4C7p", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes variational relational learning by learning relations between two inputs via variational inference on a probabilistic graphical model (PGM). The PGM that they use factors as p(a)p(z)p(b|a,z) where a,b are the two inputs and z is the supposed relationship between them. The example shown in the experiments is rotational mnist, where b is a rotated version of a, and z should encode the degree of rotation. The paper learns both the forward network and in inference network in a VAE-like approach; the elbo derivations appear correct to me.\n\nIn section 3.3 the paper describes two sources of problems with naively optimizing based on the above approach: 1) deterministic mapping where a completely determines b, and no learning of relation is necessary, and 2) information shortcut where z can completely encode b. To me 1 does not seem like a big drawback since it is more of a limitation with the underlying data. For 2, the paper describes a data-augmentation technique they call RPDA, as well as using an informative prior on z. The experimental results seem reasonable, although having setting beyond mnist would have been nice.\n\nThe two methods described to handle the information shortcut issue are not super satisfactory in my opinion. The data augmentation technique relies on designing augmentations g that preserve the relationship, so that (a,b) are related in the same way as (g(a), g(b)). But coming up with such augmentations seems to rely on us knowing properties of the relationship z, which is exactly what we are trying to learn in the first place. The alternative of using an informative prior is not super clear to me, since it seems that with a powerful enough model p(b|a,z), scaling and shifting p(z) to fit a Gaussian will not prevent the model from completely encoding b as a function of z.\n\nIn my opinion constraining the expressiveness of z seems like the right path, but instead of regularizing p(z) to be a Gaussian, I would consider using discrete z's that explicitly limits the theoretical # of bits z can store, hence preventing it from encoding b. I think this also intuitively aligns with how many relations seem more discrete than continuous (a is friend of b, a is adjective form of b, ...etc). I would be interested in seeing the experiments done using discrete z's, which seems natural too given the 5 discrete rotations considered in the paper.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a variational learning framework for inferring relations between data points. The authors further introduce novel regularizers to avoid unfavorable solutions to their relational learning problem. Qualitative results are provided on rotated versions of MNIST. Additional qualitative results on the Yale face dataset are provided in the appendix.

The reviewers agree that the idea overall is interesting, and the chosen experiments certainly provide some insight into the idea behind the method and demonstrate that the method is learning reasonable representations of relations between data points. I share the sentiment of the reviewers, however, that this paper is not yet ready for publication. The paper in its current form lacks clear positioning against related problems and related research in this community, and the experiments are all qualitative in nature without the attempt to rigorously compare the proposed method against established techniques. The argument brought forward by the authors that the proposed problem is completely novel and therefore no baselines exist is unconvincing as pointed out by the reviewers, as there is related research on e.g. spatial transformer networks [1], neural relational inference [2], and discovery of causal mechanisms [3], which similarly address the problem of discovering relations, interactions, or transformations. Even if these methods don't exactly fit the problem setup presented in this paper, an attempt should be made to design an evaluation that allows one to compare against some of these approaches, especially given that the paper claims to address the general problem of inferring relations between pairs of data points. Overall, I am confident that this would make the paper stronger and more relevant to this community.

[1] Jaderberg et al., Spatial Transformer Networks (NeurIPS 2015)
[2] Kipf et al., Neural Relational Inference for Interacting Systems (ICML 2018)
[3] Parascandolo et al., Learning Independent Causal Mechanisms (ICML 2018)

Decision: Reject